Semantic Analyses of Open-Ended Responses From Professional Development Workshop Promoting Computational Thinking in Rural Schools
Keywords:
K-12 education, rural classroom, formative assessment, teaching strategy, semantic analysis, thematic analysis, sentiment analysisAbstract
The development of curriculum and access to educational resources related to applied computing is lacking for students in K-12 schools particularly in rural areas, despite the large and growing demand for computing skills in the job market. Motivated by this need, an interdisciplinary professional development workshop was designed to promote computational thinking and curriculum integration among teachers involved in teaching core curricula including writing, math, science, and social studies in grades 3-8 in a rural midwestern state in the USA, as part of a longitudinal grant-funded program. Open-text feedback was collected before, during, and immediately after the workshop in response to multiple types of formative assessments. In this paper, we present several forms of data representation from exploratory textual analyses based on the feedback collected from the workshop participants. Semantic analysis tools including sentiment analysis and thematic analysis facilitated the identification of common themes in perception among grade 3-8 teachers relating to the implementation of computational concepts in their classrooms. Results suggest that these techniques can be useful in evaluating open-ended feedback to represent patterns of response which may aid in the identification of actionable insights related to adult learner perceptions, including interest and self-efficacy.
Downloads
References
Benotti, L., Martinez, M. C., & Schapachnik, F. (2017). A tool for introducing computer science with automatic formative assessment. IEEE Transactions on Learning Technologies, 11(2), 179-192. Retrieved from https://doi.org/10.1109/TLT.2017.2682084 DOI: https://doi.org/10.1109/TLT.2017.2682084
Bird, S. (2006, July). NLTK: The natural language toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions (pp. 69-72). Retrieved from https://aclanthology.org/P06-4018.pdf DOI: https://doi.org/10.3115/1225403.1225421
Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability (formerly: Journal of Personnel Evaluation in Education), 21(1), 5–31. Retrieved from https://doi.org/10.1007/s11092-008-9068-5 DOI: https://doi.org/10.1007/s11092-008-9068-5
Brown, E., & Brown, R. (2020). The Effect of Advanced Placement Computer Science Course Taking on College Enrollment. West Coast Analytics. Retrieved from https://www.cuny.edu/wp-content/uploads/sites/4/page-assets/about/administration/offices/oira/policy/seminars/APHighSchoolManuscript_JHRNotypset.pdf
Chamblee, G., & Slough, S. (2004). Using the concerns-based adoption model to assess changes in technology implementation. In Society for Information Technology & Teacher Education International Conference (pp. 864-871). Association for the Advancement of Computing in Education (AACE). Retrieved from https://www.learntechlib.org/primary/p/13584/
Chen, C. M., Li, M. C., & Huang, Y. L. (2020). Developing an instant semantic analysis and feedback system to facilitate learning performance of online discussion. Interactive Learning Environments, 1-19. Retrieved from https://doi.org/10.1080/10494820.2020.1839505
Chetty, R., Friedman, J. N., & Rockoff, J. E. (2014). Measuring the impacts of teachers II: Teacher value-added and student outcomes in adulthood. American Economic Review, 104(9), 2633–79. Retrieved from https://doi.org/10.1257/aer.104.9.2633 DOI: https://doi.org/10.1257/aer.104.9.2633
Clarke, V., & Braun, V. (2014). Thematic analysis. In Encyclopedia of Critical Psychology (pp. 1947–1952). Springer, New York, NY. Retrieved from https://doi.org/10.1007/978-1-4614-5583-7_311 DOI: https://doi.org/10.1007/978-1-4614-5583-7_311
Code Advocacy Coalition (2021). State of computer science education: Accelerating action through advocacy. Retrieved from https://advocacy.code.org/2021_state_of_cs.pdf
Djudin, T. (2021). Promoting students’ conceptual change by integrating the 3-2-1 reading technique with refutation text in the physics learning of buoyancy. Journal of Turkish Science Education, 18(2), 290–303. Retrieved from http://www.tused.org/index.php/tused/article/view/734/668
Education Commission of the States (2019). Stem content: Give more united states students early access to computer science. Retrieved from https://vitalsigns.ecs.org/state/united-states/curriculumprogram-grade-stats
Feldman, R. (2013). Techniques and applications for sentiment analysis. Communications of the ACM, 56(4), 82-89. Retrieved from https://doi.org/10.1145/2436256.2436274 DOI: https://doi.org/10.1145/2436256.2436274
Gabby, S., Avargil, S., Herscovitz, O., & Dori, Y. J. (2017). The case of middle and high school chemistry teachers implementing technology: Using the concerns-based adoption model to assess change processes. Chemistry Education Research and Practice, 18(1), 214-232. Retrieved from https://www.learntechlib.org/primary/p/13584/ DOI: https://doi.org/10.1039/C6RP00193A
Gottipati, S., Shankararaman, V., & Lin, J. R. (2018). Text analytics approach to extract course improvement suggestions from students’ feedback. Research and Practice in Technology Enhanced Learning, 13(1), 1-19. Retrieved from https://doi.org/10.1186/s41039-018-0073-0 DOI: https://doi.org/10.1186/s41039-018-0073-0
Graham, S., Bruch, J., Fitzgerald, J., Friedrich, L. D., Furgeson, J., Greene, K., ... & Smither Wulsin, C. (2016). Teaching Secondary Students to Write Effectively. Educator's Practice Guide. What Works Clearinghouse.™ NCEE 2017-4002. What Works Clearinghouse.
Guskey, T. R. (2005). Formative Classroom Assessment and Benjamin S. Bloom: Theory, Research, and Implications [Paper presentation]. Annual Meeting of the American Educational Research Association, Montreal, Canada. Retrieved from https://files.eric.ed.gov/fulltext/ED490412.pdf
Guzdial, M., & Hill, R. K. (2019). Getting high school, college students interested in CS. Communications of the ACM, 62(12), 10–11. Retrieved from https://doi.org/10.1145/3365581
Hamilton, W. L., Clark, K., Leskovec, J., & Jurafsky, D. (2016, November). Inducing domain-specific sentiment lexicons from unlabeled corpora. In Proceedings of the Conference on Empirical methods in Natural Language Processing. NIH Public Access. Retrieved from https://doi.org/10.18653/v1/D16-1057 DOI: https://doi.org/10.18653/v1/D16-1057
Hashemi, A., Mobini, F., & Karimkhanlooie, G. (2016). The impact of content-based pre-reading activities on iranian high school EFL learners’ reading comprehension. Journal of Language Teaching and Research, 7(1), 137. Retrieved from https://doi.org/10.17507/jltr.0701.15 DOI: https://doi.org/10.17507/jltr.0701.15
Helminski, L. (1995). Total quality in instruction: A systems approach. In H. V. Roberts (Ed.), Academic initiatives in total quality for higher education (pp. 309–362). Milwaukee, WI: ASQC Quality Press.
Hu, M., & Liu, B. (2004, August). Mining and summarizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 168-177). Retrieved from https://doi.org/10.1145/1014052.1014073 DOI: https://doi.org/10.1145/1014052.1014073
Hubwieser, P., Giannakos, M. N., Berges, M., Brinda, T., Diethelm, I., Magenheim, J., ... & Jasute, E. (2015). A global snapshot of computer science education in K-12 schools. In Proceedings of the 2015 ITiCSE on Working Group Reports (pp. 65-83). Retrieved from https://doi.org/10.1145/2858796.2858799 DOI: https://doi.org/10.1145/2858796.2858799
Lamb, R., Annetta, L., Vallett, D., Firestone, J., Schmitter-Edgecombe, M., Walker, H., ... Hoston, D. (2018). Psychosocial factors impacting STEM career selection. The Journal of Educational Research, 111(4), 446–458. Retrieved from https://doi.org/10.1080/00220671.2017.1295359 DOI: https://doi.org/10.1080/00220671.2017.1295359
Liu, J., Lin, C. H., Hasson, E. P., & Barnett, Z. D. (2011, March). Introducing computer science to K-12 through a summer computing workshop for teachers. In Proceedings of the 42nd ACM technical symposium on computer science education (pp. 389-394). Retrieved from https://doi.org/10.1145/1953163.1953277 DOI: https://doi.org/10.1145/1953163.1953277
Loughran, T., & McDonald, B. (2011). When is a liability not a liability? Textual analysis, dictionaries, and 10‐Ks. The Journal of Finance, 66(1), 35-65. Retrieved from https://doi.org/10.1111/j.1540-6261.2010.01625.x DOI: https://doi.org/10.1111/j.1540-6261.2010.01625.x
Mahadeo, J., Hazari, Z., & Potvin, G. (2020). Developing a computing identity framework: Understanding computer science and information technology career choice. ACM Transactions on Computing Education (TOCE), 20(1), 1–14. Retrieved from https://doi.org/10.1145/3365571
Marré, A. (2017). Rural education at a glance, 2017 edition. U.S. Department of Agriculture, Economic Research Service, Economic Information Bulletin, 171. Retrieved from https://www.ers.usda.gov/webdocs/publications/83078/eib-171.pdf?v=3841.6
Masood, K., Khan, M. A., Saeed, U., Al Ghamdi, M. A., Asif, M., & Arfan, M. (2022). Semantic Analysis to Identify Students’ Feedback. The Computer Journal, 65(4), 918-925. Retrieved from https://doi.org/10.1093/comjnl/bxaa130
Maxwell, J. A., & Chmiel, M. (2014). Notes toward a theory of qualitative data analysis. The SAGE Handbook of Qualitative Data Analysis, 21-34. DOI: https://doi.org/10.4135/9781446282243.n2
Mohammad, S. M., & Turney, P. D. (2013). Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3), 436-465. Retrieved from https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-8640.2012.00460.x DOI: https://doi.org/10.1111/j.1467-8640.2012.00460.x
Newlove, B. W., & Hall, G. E. (1976). A Manual for Assessing Open-Ended Statements of Concern About an Innovation. Retrieved from https://eric.ed.gov/?id=ED144207
Nielsen, F. Å. (2011, March). AFINN: A new word list for sentiment analysis on Twitter. Informatics and Mathematical Modelling, Technical University of Denmark. Retrieved from http://www2.compute.dtu.dk/pubdb/pubs/6010-full.html
Nowell, L. S., Norris, J. M., White, D. E., & Moules, N. J. (2017). Thematic analysis: Striving to meet the trustworthiness criteria. International Journal of Qualitative Methods, 16(1). Retrieved from https://doi.org/10.1177/1609406917733847 DOI: https://doi.org/10.1177/1609406917733847
Ogle, D. M. (1986). K-W-L: A Teaching Model That Develops Active Reading of Expository Text. The ReadingTeacher, 39(6), 564–570. http://www.jstor.org/stable/20199156 DOI: https://doi.org/10.1598/RT.39.6.11
Patka, M., Wallin-Ruschman, J., Wallace, T., & Robbins, C. (2016). Exit cards: creating a dialogue for continuous evaluation. Teaching in Higher Education, 21(6), 659–668. Retrieved from https://doi.org/10.1080/13562517.2016.1167033 DOI: https://doi.org/10.1080/13562517.2016.1167033
Salehi, S., Wang, K. D., Toorawa, R., & Wieman, C. (2020, February). Can majoring in computer science improve general problem-solving skills? In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (pp. 156–161). Retrieved from https://doi.org/10.1145/3328778.3366808
Showalter, D., Klein, R., Johnson, J., & Hartman, S. L. (2017). Why Rural Matters 2015-2016: Understanding the Changing Landscape. A Report of the Rural School and Community Trust. Rural School and Community Trust. Retrieved from https://files.eric.ed.gov/fulltext/ED590169.pdf
Steele, J., & Dyer, T. (2014). Use of KWLS in the online classroom as it correlates to increased participation. Journal of Instructional Research, 3, 8–14. Retrieved from https://files.eric.ed.gov/fulltext/EJ1127637.pdf
U.S. Bureau of Labor Statistics (2021, May). Occupational Outlook Handbook: Computer and information technology occupations. Retrieved from https://www.bls.gov/ooh/computer-and-information-technology/home.htm
Wang, J., & Hejazi Moghadam, S. (2017, March). Diversity barriers in K-12 computer science education: Structural and social. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education (pp. 615-620). Retrieved from https://doi.org/10.1145/3017680.3017734 DOI: https://doi.org/10.1145/3017680.3017734
Published
How to Cite
Issue
Section
Copyright (c) 2023 Amber Gillenwaters, Razib Iqbal, Diana Piccolo, Tammi Davis, Keri Franklin, David Cornelison, Judith Martinez, Andrew Homburg, Julia Cottrell, Melissa Page
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
