Bad Tools Hurt: Lessons for teaching computer security skills to undergraduates
Keywords:
security, coding, teaching, usability, toolsAbstract
Understanding why developers continue to misuse security tools is critical to designing safer software, yet the underlying reasons developers fail to write secure code are not well understood. In order to better understand how to teach these skills, we conducted two comparatively large-scale usability studies with undergraduate CS students to assess factors that affect success rates in securing web applications against cross-site request forgery (CSRF) attacks. First, we examined the impact of providing students with example code and/or a testing tool. Next, we examined the impact of working in pairs. We found that access to relevant secure code samples gave significant benefit to security outcomes. However, access to the tool alone had no significant effect on security outcomes, and surprisingly, the same held true for the tool and example code combined. These results confirm the importance of quality example code and demonstrate the potential danger of using security tools in the classroom that have not been validated for usability. No individual differences predicted one’s ability to complete the task. We also found that working in pairs had a significant positive effect on security outcomes. These results provide useful directions for teaching computer security programming skills to undergraduate students.
Downloads
Download data is not yet available.
References
Acar, Y., Backes, M., Fahl, S., Garfinkel, S., Kim, D., Mazurek, M. L., and Stransky, C. (2017). Comparing the Usability of Cryptographic APIs. In 2017 IEEE Symposium on Security and Privacy (SP), pages 154-171.
https://doi.org/10.1109/SP.2017.52
Acar, Y., Backes, M., Fahl, S., Kim, D., Mazurek, M. L., and Stransky, C. (2016). You Get Where You're Looking for: The Impact of Information Sources on Code Security. In 2016 IEEE Symposium on Security and Privacy (SP), pages 289-305. https://doi.org/10.1109/SP.2016.25
Acar, Y., Stransky, C., Wermke, D., Mazurek, M. L., and Fahl, S. (2017). Security Developer Studies with GitHub Users: Exploring a Convenience Sample. In Thirteenth Symposium on Usable Privacy and Security (SOUPS 2017), pages 81-95, Santa Clara, CA. USENIX Association. https://www.usenix.org/system/files/conference/soups2017/soups2017-acar.pdf
Arisholm, E., Gallis, H., Dyba, T., and Sjoberg, D. I. K. (2007). Evaluating pair programming with respect to system complexity and programmer expertise. IEEE Transactions on Software Engineering, 33(2):65-86.
https://doi.org/10.1109/TSE.2007.17
Bangor, A., Kortum, P., and Miller, J. (2009). Determining what individual sus scores mean: Adding an adjective rating scale. Journal of Usability Studies, 4(3):114-123. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.177.1240&rep=rep1&type=pdf
Bangor, A., Kortum, P. T., and Miller, J. T. (2008). An empirical evaluation of the system usability scale. International Journal of Human-Computer Interaction, 24(6):574-594. https://doi.org/10.1080/10447310802205776
Begel, A. and Nagappan, N. (2008). Pair programming: What's in it for me? In Proceedings of the Second ACMIEEE International Symposium on Empirical Software Engineering and Measurement, ESEM '08, page 120-128, New York, NY, USA. Association for Computing Machinery. https://doi.org/10.1145/1414004.1414026
Bernhard, M., Sharman, J., Acemyan, C. Z., Kortum, P., Wallach, D. S., and Halderman, J. A. (2019). On the Usability of HTTPS Deployment. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI '19, pages 310:1-310:10, New York, NY, USA. ACM. https://doi.org/10.1145/3290605.3300540
Brooke, J. (1996). SUS: A quick and dirty usability scale. Usability evaluation in industry, page 189.
Eierman, M. and Iversen, J. (2018). Comparing testdriven development and pair programming to improve the learning of programming languages. Journal of the Midwest Association for Information Systems, 2018:23-36.
https://doi.org/10.17705/3jmwa.000038
Fischer, F., Böttinger, K., Xiao, H., Stransky, C., Acar, Y., Backes, M., and Fahl, S. (2017). Stack Overflow Considered Harmful? The Impact of Copy Paste on Android Application Security. In 2017 IEEE Symposium on Security and Privacy, pages 121-136. https://doi.org/10.1109/SP.2017.31
Gorski, P. L., Iacono, L. L., Wermke, D., Stransky, C., Moeller, S., Acar, Y., and Fahl, S. (2018). Developers Deserve Security Warnings, Too: On the Effect of Integrated Security Advice on Cryptographic API Misuse. In SOUPS @ USENIX Security Symposium. https://www.usenix.org/system/files/conference/soups2018/soups2018-gorski.pdf
Hanks, B., Mcdowell, C., Draper, D., and Krnjajic, M. (2004). Program quality with pair programming in cs1. In ACM Sigcse Bulletin,(36), pages 176-180. https://doi.org/10.1145/1026487.1008043
Hannay, J., Dybå, T., Arisholm, E., and Sjøberg, D. (2009). The effectiveness of pair programming: A meta-analysis. Information and Software Technology, 51:1110-1122. https://doi.org/10.1016/j.infsof.2009.02.001
ISO 9241-11 (2018). Ergonomics of human-system interaction - Part 11: Usability: Definitions and concepts.
Kortum, P. and Acemyan, C. Z. (2013). How low can you go?: Is the system usability scale range restricted? Journal of Usability Studies, 9(1):14-24. http://tecfa.unige.ch/tecfa/maltt/ergo/1415/UtopiaPeriode4/articles/JUS_Kortum_November_2013.pdf
Krombholz, K., Mayer, W., Schmiedecker, M., and Weippl, E. (2017). 'I Have No Idea What I'm Doing' - On the Usability of Deploying HTTPS. In 26th USENIX Security Symposium (USENIX Security 17), pages 1339-1356, Vancouver, BC. USENIX Association. https://www.usenix.org/system/files/conference/usenixsecurity17/sec17-krombholz.pdf
Mcdowell, C., Werner, L., Bullock, H., and Fernald, J. (2002). The effects of pair-programming on performance in an introductory programming course. In ACM SIGCSE Bulletin,(34), pages 38-42. https://doi.org/10.1145/563340.563353
Mcdowell, C., Werner, L., Bullock, H. E., and Fernald, J. (2003). The impact of pair programming on student performance, perception and persistence. In 25th International Conference on Software Engineering, 2003, pages 602-607.
https://doi.org/10.1109/ICSE.2003.1201243
McDowell, C., Werner, L., Bullock, H. E., and Fernald, J. (2006). Pair programming improves student retention, confidence, and program quality. Communications of the ACM, 49(8):90-95. https://doi.org/10.1145/1145287.1145293
Mindermann, K. and Wagner, S. (2018). Usability and Security Effects of Code Examples on Crypto APIs - CryptoExamples: A platform for free, minimal, complete and secure crypto examples. 2018 16th Annual Conference on Privacy, Security and Trust (PST). https://doi.org/10.1109/PST.2018.8514203
Naiakshina, A., Danilova, A., Gerlitz, E., and Smith, M. (2020). On conducting security developer studies with cs students: Examining a password-storage study with cs students, freelancers, and company developers. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1-13. https://doi.org/10.1145/3313831.3376791
Naiakshina, A., Danilova, A., Gerlitz, E., von Zezschwitz, E., and Smith, M. (2019). 'If You Want, I Can Store the Encrypted Password': A Password-Storage Field Study with Freelance Developers. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI '19, pages 140:1-140:12, New York, NY, USA. ACM.
https://doi.org/10.1145/3290605.3300370
Naiakshina, A., Danilova, A., Tiefenau, C., and Smith, M. (2018). Deception task design in developer password studies: Exploring a student sample. In Fourteenth Symposium on Usable Privacy and Security (SOUPS 2018), pages 297-313, Baltimore, MD. USENIX Association. https://www.usenix.org/system/files/conference/soups2018/soups2018-naiakshina.pdf
Oliveira, D. S., Lin, T., Rahman, M. S., Akefirad, R., Ellis, D., Perez, E., Bobhate, R., DeLong, L. A., Cappos, J., and Brun, Y. (2018). API Blindspots: Why Experienced Developers Write Vulnerable Code. In Fourteenth Symposium on Usable Privacy and Security (SOUPS 2018), pages 315- 328, Baltimore, MD. USENIX Association. https://www.usenix.org/system/files/conference/soups2018/soups2018-oliveira.pdf
Patitsas, E., Berlin, J., Craig, M., and Easterbrook, S. (2016). Evidence that computer science grades are not bimodal. In ICER '16: Proceedings of the 2016 ACM Conference on International Computing Education Research, pages 113-121. https://doi.org/10.1145/2960310.2960312
Smith, M. O., Giugliano, A., and DeOrio, A. (2018). Long term effects of pair programming. IEEE Transactions on Education, 61(3):187-194. https://doi.org/10.1109/TE.2017.2773024
Ukrop, M. and Matyas, V. (2018). Why Johnny the Developer Can't Work with Public Key Certificates - An Experimental Study of OpenSSL Usability. In CT-RSA, volume 10808 of Lecture Notes in Computer Science, pages 45-64. Springer. https://doi.org/10.1007/978-3-319-76953-0_3
Weber, S., Coblenz, M., Myers, B., Aldrich, J., and Sunshine, J. (2017). Empirical studies on the security and usability impact of immutability. In 2017 IEEE Cybersecurity Development (SecDev), pages 50-53. https://doi.org/10.1109/SecDev.2017.21
Werner, L., Hanks, B., and Mcdowell, C. (2004). Pair programming helps female computer science students. ACM Journal of Educational Resources in Computing, 4. https://doi.org/10.1145/1060071.1060075
Williams, L., Kessler, R. R., Cunningham, W., and Jeffries, R. (2000). Strengthening the case for pair programming. IEEE Software, 17(4):19-25. https://doi.org/10.1109/52.854064
Williams, L. and Upchurch, R. (2001). In support of student pair-programming. In ACM SIGCSE Bulletin,(33), pages 327-331. https://doi.org/10.1145/366413.364614
Williams, L., Wiebe, E., Yang, K., Ferzli, M., and Miller, C. (2002). In support of pair programming in the introductory computer science course. Computer Science Education, 12(3):197-212. https://doi.org/10.1076/csed.12.3.197.8618
Zeier, A., Wiesmaier, A., and Heinemann, A. (2019). API Usability of Stateful Signature Schemes, pages 221-240. Springer.
https://doi.org/10.1007/978-3-030-26834-3_13
https://doi.org/10.1109/SP.2017.52
Acar, Y., Backes, M., Fahl, S., Kim, D., Mazurek, M. L., and Stransky, C. (2016). You Get Where You're Looking for: The Impact of Information Sources on Code Security. In 2016 IEEE Symposium on Security and Privacy (SP), pages 289-305. https://doi.org/10.1109/SP.2016.25
Acar, Y., Stransky, C., Wermke, D., Mazurek, M. L., and Fahl, S. (2017). Security Developer Studies with GitHub Users: Exploring a Convenience Sample. In Thirteenth Symposium on Usable Privacy and Security (SOUPS 2017), pages 81-95, Santa Clara, CA. USENIX Association. https://www.usenix.org/system/files/conference/soups2017/soups2017-acar.pdf
Arisholm, E., Gallis, H., Dyba, T., and Sjoberg, D. I. K. (2007). Evaluating pair programming with respect to system complexity and programmer expertise. IEEE Transactions on Software Engineering, 33(2):65-86.
https://doi.org/10.1109/TSE.2007.17
Bangor, A., Kortum, P., and Miller, J. (2009). Determining what individual sus scores mean: Adding an adjective rating scale. Journal of Usability Studies, 4(3):114-123. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.177.1240&rep=rep1&type=pdf
Bangor, A., Kortum, P. T., and Miller, J. T. (2008). An empirical evaluation of the system usability scale. International Journal of Human-Computer Interaction, 24(6):574-594. https://doi.org/10.1080/10447310802205776
Begel, A. and Nagappan, N. (2008). Pair programming: What's in it for me? In Proceedings of the Second ACMIEEE International Symposium on Empirical Software Engineering and Measurement, ESEM '08, page 120-128, New York, NY, USA. Association for Computing Machinery. https://doi.org/10.1145/1414004.1414026
Bernhard, M., Sharman, J., Acemyan, C. Z., Kortum, P., Wallach, D. S., and Halderman, J. A. (2019). On the Usability of HTTPS Deployment. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI '19, pages 310:1-310:10, New York, NY, USA. ACM. https://doi.org/10.1145/3290605.3300540
Brooke, J. (1996). SUS: A quick and dirty usability scale. Usability evaluation in industry, page 189.
Eierman, M. and Iversen, J. (2018). Comparing testdriven development and pair programming to improve the learning of programming languages. Journal of the Midwest Association for Information Systems, 2018:23-36.
https://doi.org/10.17705/3jmwa.000038
Fischer, F., Böttinger, K., Xiao, H., Stransky, C., Acar, Y., Backes, M., and Fahl, S. (2017). Stack Overflow Considered Harmful? The Impact of Copy Paste on Android Application Security. In 2017 IEEE Symposium on Security and Privacy, pages 121-136. https://doi.org/10.1109/SP.2017.31
Gorski, P. L., Iacono, L. L., Wermke, D., Stransky, C., Moeller, S., Acar, Y., and Fahl, S. (2018). Developers Deserve Security Warnings, Too: On the Effect of Integrated Security Advice on Cryptographic API Misuse. In SOUPS @ USENIX Security Symposium. https://www.usenix.org/system/files/conference/soups2018/soups2018-gorski.pdf
Hanks, B., Mcdowell, C., Draper, D., and Krnjajic, M. (2004). Program quality with pair programming in cs1. In ACM Sigcse Bulletin,(36), pages 176-180. https://doi.org/10.1145/1026487.1008043
Hannay, J., Dybå, T., Arisholm, E., and Sjøberg, D. (2009). The effectiveness of pair programming: A meta-analysis. Information and Software Technology, 51:1110-1122. https://doi.org/10.1016/j.infsof.2009.02.001
ISO 9241-11 (2018). Ergonomics of human-system interaction - Part 11: Usability: Definitions and concepts.
Kortum, P. and Acemyan, C. Z. (2013). How low can you go?: Is the system usability scale range restricted? Journal of Usability Studies, 9(1):14-24. http://tecfa.unige.ch/tecfa/maltt/ergo/1415/UtopiaPeriode4/articles/JUS_Kortum_November_2013.pdf
Krombholz, K., Mayer, W., Schmiedecker, M., and Weippl, E. (2017). 'I Have No Idea What I'm Doing' - On the Usability of Deploying HTTPS. In 26th USENIX Security Symposium (USENIX Security 17), pages 1339-1356, Vancouver, BC. USENIX Association. https://www.usenix.org/system/files/conference/usenixsecurity17/sec17-krombholz.pdf
Mcdowell, C., Werner, L., Bullock, H., and Fernald, J. (2002). The effects of pair-programming on performance in an introductory programming course. In ACM SIGCSE Bulletin,(34), pages 38-42. https://doi.org/10.1145/563340.563353
Mcdowell, C., Werner, L., Bullock, H. E., and Fernald, J. (2003). The impact of pair programming on student performance, perception and persistence. In 25th International Conference on Software Engineering, 2003, pages 602-607.
https://doi.org/10.1109/ICSE.2003.1201243
McDowell, C., Werner, L., Bullock, H. E., and Fernald, J. (2006). Pair programming improves student retention, confidence, and program quality. Communications of the ACM, 49(8):90-95. https://doi.org/10.1145/1145287.1145293
Mindermann, K. and Wagner, S. (2018). Usability and Security Effects of Code Examples on Crypto APIs - CryptoExamples: A platform for free, minimal, complete and secure crypto examples. 2018 16th Annual Conference on Privacy, Security and Trust (PST). https://doi.org/10.1109/PST.2018.8514203
Naiakshina, A., Danilova, A., Gerlitz, E., and Smith, M. (2020). On conducting security developer studies with cs students: Examining a password-storage study with cs students, freelancers, and company developers. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1-13. https://doi.org/10.1145/3313831.3376791
Naiakshina, A., Danilova, A., Gerlitz, E., von Zezschwitz, E., and Smith, M. (2019). 'If You Want, I Can Store the Encrypted Password': A Password-Storage Field Study with Freelance Developers. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI '19, pages 140:1-140:12, New York, NY, USA. ACM.
https://doi.org/10.1145/3290605.3300370
Naiakshina, A., Danilova, A., Tiefenau, C., and Smith, M. (2018). Deception task design in developer password studies: Exploring a student sample. In Fourteenth Symposium on Usable Privacy and Security (SOUPS 2018), pages 297-313, Baltimore, MD. USENIX Association. https://www.usenix.org/system/files/conference/soups2018/soups2018-naiakshina.pdf
Oliveira, D. S., Lin, T., Rahman, M. S., Akefirad, R., Ellis, D., Perez, E., Bobhate, R., DeLong, L. A., Cappos, J., and Brun, Y. (2018). API Blindspots: Why Experienced Developers Write Vulnerable Code. In Fourteenth Symposium on Usable Privacy and Security (SOUPS 2018), pages 315- 328, Baltimore, MD. USENIX Association. https://www.usenix.org/system/files/conference/soups2018/soups2018-oliveira.pdf
Patitsas, E., Berlin, J., Craig, M., and Easterbrook, S. (2016). Evidence that computer science grades are not bimodal. In ICER '16: Proceedings of the 2016 ACM Conference on International Computing Education Research, pages 113-121. https://doi.org/10.1145/2960310.2960312
Smith, M. O., Giugliano, A., and DeOrio, A. (2018). Long term effects of pair programming. IEEE Transactions on Education, 61(3):187-194. https://doi.org/10.1109/TE.2017.2773024
Ukrop, M. and Matyas, V. (2018). Why Johnny the Developer Can't Work with Public Key Certificates - An Experimental Study of OpenSSL Usability. In CT-RSA, volume 10808 of Lecture Notes in Computer Science, pages 45-64. Springer. https://doi.org/10.1007/978-3-319-76953-0_3
Weber, S., Coblenz, M., Myers, B., Aldrich, J., and Sunshine, J. (2017). Empirical studies on the security and usability impact of immutability. In 2017 IEEE Cybersecurity Development (SecDev), pages 50-53. https://doi.org/10.1109/SecDev.2017.21
Werner, L., Hanks, B., and Mcdowell, C. (2004). Pair programming helps female computer science students. ACM Journal of Educational Resources in Computing, 4. https://doi.org/10.1145/1060071.1060075
Williams, L., Kessler, R. R., Cunningham, W., and Jeffries, R. (2000). Strengthening the case for pair programming. IEEE Software, 17(4):19-25. https://doi.org/10.1109/52.854064
Williams, L. and Upchurch, R. (2001). In support of student pair-programming. In ACM SIGCSE Bulletin,(33), pages 327-331. https://doi.org/10.1145/366413.364614
Williams, L., Wiebe, E., Yang, K., Ferzli, M., and Miller, C. (2002). In support of pair programming in the introductory computer science course. Computer Science Education, 12(3):197-212. https://doi.org/10.1076/csed.12.3.197.8618
Zeier, A., Wiesmaier, A., and Heinemann, A. (2019). API Usability of Stateful Signature Schemes, pages 221-240. Springer.
https://doi.org/10.1007/978-3-030-26834-3_13
Published
2021-12-17
How to Cite
Sharman, J., Acemyan, C., Kortum, P., & Wallach, D. (2021). Bad Tools Hurt: Lessons for teaching computer security skills to undergraduates. International Journal of Computer Science Education in Schools, 5(2), 74–92. https://doi.org/10.21585/ijcses.v5i2.131
Issue
Section
Full Papers
Copyright & Licensing
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
