The Development and Validation of the Programming Anxiety Scale

The main goal of the current study is to develop a reliable instrument to measure programming anxiety in university students. A pool of 33 items based on extensive literature review and experts' opinions were created by researchers. The draft scale comprised three factors applied to 392 university students from two different universities in Turkey for exploratory factor analysis. The number and character of the underlying components in the scale were determined using exploratory factor analysis. After exploratory factor analysis, confirmatory factor analysis was conducted on the draft scale using a sample of 295 university students. Confirmatory factor analysis was carried out to ensure that the data fit the retrieved factor structure. The internal consistency coefficient (Cronbach's alpha) was calculated for the full scale and each dimension for reliability analysis. For convergent validity, the factor loading of the indicator, the average variance extracted, composite reliability, and maximum share variance values were calculated. Additionally, convergent validity was tested through (1) comparison of mean values of factors and total programming anxiety depending on gender and (2) correlation analysis of factors, total programming anxiety, and course grade of students. The Fornell & Larcker criterion and the Heterotrait-Monotrait correlation ratio were utilized to assess discriminant validity. According to analysis results, the Programing Anxiety Scale (PAS) comprised 11 items in two factors: classmates and self-confidence. Similarly, results revealed that The PAS has good psychometric properties and can be used to assess the programming anxiety of university students.


Introduction
With the development of the internet and mobile technologies in the last two decades, breakthroughs have been experienced in many fields of computer science such as big data, artificial intelligence, blockchain, bioinformatics, wearable technologies, cloud computing, 3D printers, robotics, and virtual reality. Responsibilities of computer science such as automating the processes, facilitating communication, providing better products and services, assisting the world to be more productive have caused human beings to be more dependent on software (Santos, Tedesco, Borba, & Brito, 2020). As a result, all developed and developing countries are required to raise qualified individuals who can maintain the software used and produce practical solutions to new problems encountered in the future (Demirer & Sak, 2016). One of the conditions for the success of this task is to provide students with programming skills, which is considered one of the requirements of being a well-educated and knowledgeable citizen (Al-Makhzoomy, 2018;Kert & Uğraş, 2009). However, according to several studies, most computer science students regard programming courses as complicated and intimidating (Bennedsen & Caspersen, 2007;Connolly, Murphy, & Moore, 2009;Jenkins, 2002;Owolabi, Olanipekun, & Iwerima, 2014;Robins, Rountree, & Rountree, 2003;Wiedenbeck, Labelle, & Kain, 2004). Moreover, studies show that programming courses have high dropout and failure rates (Bennedsen & Caspersen, 2007;Luxton-Reilly et al., 2019).
Since learners' self-belief plays a fundamental role in intellectual development (Berland & Lee, 2011;Pajares, 1992), Jiang, Zhao, Wang, and Hu (2020) believe that this trauma happens when students lose their self-efficacy in programming, which negatively affects learning outcomes. Connolly et al. (2007) propose a cognitive model to explain how programming anxiety influences students' emotional, behavioral and physiological reactions (see Figure 1). The mental model asserts that students' automatic thoughts are activated in programming situations, directly influenced by their core and intermediate beliefs. Eventually, automatic thoughts affect their emotional, behavioral, and physiological reactions. According to Connolly et al. (2007), a fear of programming may commence caused by core beliefs for a student sensitive to programming anxiety. Then, intermediate thoughts of students could emerge as a fear of what other students might think about their performance and ability. Finally, automatic thoughts arise in programming situations and trigger negative thoughts and reactions.
In addition to the cognitive model for programming anxiety, Rogerson and Scott (2010) also depict an iceberg model to explain factors affecting fear of programming. According to the iceberg model, the fear of programming is induced due to the nature of programming. Rogerson and Scott (2010) cite those internal factors such as motivation, attitude, self-efficacy, and attribution often have a part in building negative perceptions of programming. At the same time, peers, teaching methodology, timing, lectures, and tutors constitute external factors.  (Connolly et al., 2007) International Journal of Computer Science Education in Schools, April 2022, Vol. 5, No. 3 ISSN 2513 The number of studies on programming anxiety has risen dramatically in the last decade. Some of these studies examined factors associated with programming anxiety, while others investigated the impact of programming anxiety on student performance and motivation. According to S Sinožić and Orehovaki (2018), the absence of programming experience, fear of programming, and a misperception of programming languages as very complex are all powerful determinants of programming anxiety among novices. Similarly, unfamiliar subjects in programming courses make students avoid programming, and programming makes them feel uncomfortable (Olipas, Leona, Villegas, Cunanan & Javate, 2021). According to studies, learners' programming anxiety levels aggregated as they were presented to programming concepts and principles. (Campbell, 2018;Dasuki & Quaye, 2016).
Several studies have also connected programming anxiety to academic performance, perceived self-efficacy, encountering errors when developing programs, gender, peers, test anxiety, mathematics, and computer anxiety. For example, Olipas et al. (2021) found a negative association between participants' academic performance and programming anxiety in a study of 348 students. Hsu and Gainsburg (2021) and Wilfong (2006) explain that self-efficacy plays a vital role in performance in programming courses, and self-efficacy has a mediating effect on the relationship between anxiety and performance. Results of a systematic review of the literature conducted by Nolan and Bergin (2016) illustrate correlates of programming anxiety as programming as a subject, test anxiety, computer anxiety (volume of computer usage), and using mathematics frequently in coding. Additionally, the students' incapacity to debug their programs increase their programming anxiety (Dasuki & Quaye, 2016;Nolan & Bergin, 2016).
Some researchers mention the effects of peers on programming anxiety. According to Nolan and Bergin (2016), when programming students learn to program in a laboratory with many peers, this circumstance can be stressful. Falkner, Falkner, and Vivian (2013) explored how collaborative practices in programming courses can cause fear and tension in learners. They concluded that working in groups prevented students from feeling comfortable in classes. There are also studies in the programming literature on the effects of gender on programming anxiety. According to Olipas and Luciano's (2020) study, female students show more programming anxiety than male students. Chang (2005) also explored a possible association between the perceived complexity of programming tasks and programming anxiety with 307 participants. According to the findings, there was a strong association between these two variables, indicating that as the perceived complexity of programming assignments increased, so did students' perceived programming anxiety levels.
Many studies in the literature state that programming anxiety is one of the factors that cause students to fail and lose interest in programming courses. It is reported that programming anxiety is critical in determining students' success in a programming course (Connolly et al., 2007;Figueroa & Amoloza, 2015;Kinnunen & Malmi, 2006;Nolan, Bergin & Mooney, 2019;Owolabi et al., 2014;Scott, 2015). With self-beliefs being the case, Kinnunen and Simon (2012) assert that learners' self-beliefs are developed due to the experiences students have while they engage in programming activities rather than the resulting quality of the programs they write. As a consequence of negative experiences and self-appraisals, learners lack the time or have no motivation to program (Kinnunen & Malmi, 2006;Scott, 2015). Similarly, Maguire et al. (2017) assert that programming anxiety causes a lack of confidence and plays a crucial role in discouraging students from carrying out programming independently.
Results of the study of Özmen and Altun (2014) show that while students with a low level of programming anxiety spend extra time on programming and code more qualified programs, students with a high level of anxiety devote limited time on programming practices and avoid learning programming. Similar results have been cited by Scott (2015), concluding that programming anxiety inhibits time spent practicing programming and decreases course participation (Bergin & Reilly, 2005). Scott and Ghinea (2014) investigated the possible adverse effects of programming anxiety on students' programming practice. Participants of the study were 239 university students. The findings revealed that students are frequently concerned when undertaking debugging activities.
In the light of all these studies in the literature, it is essential to measure the programming anxiety levels of students with reliable and valid instruments to determine students' anxiety levels and help learners overcome their anxiety and frustration in programming courses. However, despite anxiety's critical role in programming, research on anxiety scale development has been deficient. Information about these measurement instruments is summarized in Table 1. As presented in Table 1, three scales are prepared to measure primarily programming anxiety. All of the scales are based on self-reported data. In addition to these scales, it was noted that computer anxiety or information technology (IT) anxiety scales were adapted for measuring programming anxiety in several studies (see Olipas & Luciano, 2020;Scott &Ghinea, 2014, andOrehovacki, Radosevic &Konecki, 2012). Furhermore, Demir (2021) recently adapted Choo and Cheung's (1991) programming anxiety scale into Turkish.

Purpose of the Study
Studies show that reducing anxiety can enhance academic performance and achievement (Hattie, 2008). The same is true when it comes to improving the efficiency of programming courses. It is vital to identify learners' programming anxiety and work closely with students with high anxiety to develop the learning outcomes of programming courses at the highest level. In this sense, there is a need for reliable measurement tools designed to measure programming anxiety to make meaningful conclusions from the analysis. As a result, the current research aims to create a proper and reliable tool to measure programming anxiety in university students.

Method
The Computer Programming Anxiety Scale was developed and validated in three phases, illustrated in Figure 2.
In summary, dimensions of the draft scale were identified, and the item pool was generated in the first phase. In the second phase, content and phase validity were assessed. In the last stage, exploratory and confirmatory factor analysis was conducted, and construct validity was evaluated.

Phase 1: Identifying dimensions & item generation
Clark and Watson (1995) recommend beginning scale development by clearly conceptualizing the target construct and clarifying its breadth and scope. The researchers conducted a comprehensive literature review and content analysis to identify different dimensions of programming anxiety. With this respect, models and explanations related to programming anxiety were examined to develop a clear conceptualization. Furthermore, related constructs including computer anxiety, math anxiety, and test anxiety were investigated. The Computer Programming Anxiety Scale (Choo & Cheung, 1991) was used to identify programming anxiety dimensions. At the same time, scales developed to measure students' anxiety, such as programming anxiety, computer anxiety, test anxiety, math anxiety, and foreign language learning anxiety, were investigated. Depending on the studies on programming anxiety, three dimensions were proposed: (1) classmates, (2) self-confidence, and (3) errors. The "Classmates" subscale measures students' anxiety in the presence of more proficient students. The "Programming confidence" subscale measures students' feelings of inadequacy while programming. "Errors" subscale measures students' anxiety when confronted with errors during programming.
Next, a pool of 33 items was constructed to capture negative emotions during program development and debugging. The rationale underpinning including as many items as possible in the draft scale was that the number of items at the start should be twice as numerous as the final scale (Nunnally, 1994). To obtain precise and unambiguous items that reflect the specified conceptual definitions, item wording rules suggested by Carpenter (2018) were applied. This cyclical item development procedure yielded a total of 33 items, each rated on a 5-point scale from 1 ("never true") to 5 ("always true"). In this regard, "seldom true" was scored as 2, "sometimes true" was 3, and "often true" was 4.

Phase 2: Development of the scale 2.2.1 Content validity
Content validity of the draft scale was tested by interviewing five experts, three of whom were from the field of instructional technology, one from the field of Turkish language, and one from the field of psychological counseling and guidance. An expert opinion form was created in this phase. The experts were requested to rate each scale item using this form on a four-point rating scale (1 = not relevant; 2 = item requires so much revision that it is no longer relevant; 3 = item is suitable but needs minor changes; 4 = highly relevant). Data gathered from the expert opinion was used to quantify the content validity process and calculate Content Validity Index (I-CVI; Polit, Beck, & Owen, 2007). The I-CVI was calculated by dividing the experts who provided a 3 or 4 by the total number of experts for each item (Lynn, 1986). Nine items with an I-CVI-score of less than one were excluded from the draft scale using Lynn's (1986) criteria. In addition, based on the experts' recommendations, two of the retained items were revised to simplify the language. This operation yielded 24 items in the final pool (classmates: 8 items; self-confidence: 9 items; errors: 7 items). Table 2 depicts the item pool on the draft scale.

Face validity
The questionnaire's face validity was assessed quantitatively. To evaluate the qualitative face validity, nine college students enrolled in a programming course were interviewed face to face and participants rated the items based on clarity and relevancy. Despite some minor errors, all of the interviewees concurred on the clarity and comprehensibility of all of the items.

Translation of the Scale
The Programming Anxiety Scale items were created and written in Turkish at first. The data were collected utilizing this original scale. The translation of the original scale to English was carried out after the data collection process. A mixed translation strategy utilizing the back-translation method and the committee approach that was distinct from Jones, Lee, Phillips, Zhang & Jaceldo (2001) was used in the translation process. The researchers initially translated each item in the original version into English. Next, three Turkish/English bilingual professors thoroughly inspected each translated item. With the help of the multilingual teacher group, any necessary modifications to problematic items were performed. The bilingual experts agreed on the translated and original versions of the scale. Item8 I am concerned about not being able to write a program and being ridiculed by my classmates.

Self-confidence
Item9 I think I do not understand programming well.

Item10
It makes me anxious to feel that I memorize programming topics instead of learning the logic.

Item11
It makes me anxious to feel that I quickly forget what I have learned in programming lessons.

Item12
I have doubts about creating the steps (algorithm) necessary for the solution while coding the program.

Item13
I have concerns about my programming abilities.

Item14
I feel confused when the program lines become complicated.
Item15 I don't trust myself in writing programs.

Item16
I get nervous when we talk about programming.

Item17
It scares me that there are too many topics to learn in the programming lesson.

Errors
Item18 I get worried when I can not understand error messages.

Item19
The number of errors in my program makes me worried.

Item20
I am worried about encountering errors in my programs.

Item21
I feel worried when my program fails to run.

Item22
Debugging programs is a major worry for me.

Item23
I get worried about debugging my software over and over again.

Item24
It makes me worried to think that my codes will have bugs. The draft scale was validated primarily in two phases. Since using the same data set for exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) is not generally accepted as the correct method in the literature (Fokkema & Greiff, 2017), data were collected from two distinct sample groups of university students (first and second sample). While the data from the first sample was used to investigate the scale's underlying factor structure in the EFA phase, the data from the second sample was used in the CFA phase to cross-validate the EFA results. All subjects were recruited using a convenience sampling method.

Analytical Strategy
The PAS was validated using Boateng, Neilands, Frongillo, Melgar-Quiñonez, & Young's (2018) scale development recommendations. The number and character of the underlying components in the scale were determined using EFA, which CFA followed to ensure that the data fit the retrieved factor structure. The construct validity was then examined after a reliability analysis. The Kaiser-Meyer-Olkin Measure of Sampling Adequacy (KMO) and Bartlett's test were implemented to assess the adequacy of the study group (Tabachnick & Fidell, 2001). The tests showed that the data was suitable for EFA. In addition, considering Hair, Black, Babin, & Anderson's (2010) suggestions on the cutoff value for factor loadings and commonalities, these values were determined as .50 and .30, respectively. Items that loaded only one factor without any cross-loadings were kept. The research employed SPSS 22 software for EFA. For CFA, Maximum likelihood estimation was adopted to calculate the structure parameters using AMOS 22 software.

Exploratory Factor Analysis
The data's appropriateness for factor analysis was assessed before doing EFA. For this reason, Mahalanobis distance values for probable multivariate outliers were determined. Thirty-nine instances were eliminated from the analysis because their Mahalanobis values were above the necessary chi-square value of 54.05 (df = 26, alpha=.001) (Pallant, 2007). In addition, KMO was used to determine the adequacy of the sample size, and Bartlett's Test of Sphericity was used to determine whether the datum was suitable for factor analysis. The KMO sampling adequacy metric was .951, higher than the acceptable value of .60. Barlett's Test of Sphericity was also statistically significant (x 2 =3790.593, p=.000), demonstrating that the data was considerably factorable (Pallant, 2007).
In the first stage of EFA, principal axis factoring was employed for 24 items with direct oblimin rotation. The direct oblimin rotation approach was chosen because of the associated factors (Costello & Osborne, 2005;Gorsuch, 1983). After the EFA process, the correlation matrix was investigated for multicollinearity issues. Correlations in the .80's or .90's (Field, 2018) were examined, and six items (Item18, Item19, Item20, Item21, Item23, and Item24) in the "errors" factor with a correlation coefficient greater than .8 were excluded from the scale. Next, EFA was conducted again for the remaining 18 items. Four items (Item2 and Item6 in classmates and Item13 and Item16 in self-confidence) with high factor loadings in multiple factors were excluded (Burns & Machin, 2009;Howard, 2016). EFA was executed with the remaining 14 items. Based on the cutoffs for the eigenvalues and inspection of the scree plot, a two-factor model was identified that explained 69.4% of the variance in programming anxiety. The two-factor model was confirmed by a parallel analysis with 5000 randomly generated data matrices through Parallel Analysis Web Application (Patil, Surendra, Sanjay, & Donavan, 2017). These factors were labeled: (1) Classmates and (2) Self-Confidence. Table 3 includes the complete list of factor loadings. Internal consistency reliabilities (i.e., Cronbach's alpha coefficient) for the full scale and the subscales were .95, .90, and .94, respectively. Debugging programs is a major worry for me. .822

Confirmatory Factor Analysis
The 14-item Programming Anxiety Scale's two-factor model was subjected to CFA using second sample data. Before the analysis, data were screened for missing values and outliers. With this respect, 13 respondents were detected unengaged in the scale evidence by getting the same response for every item. Thus, 14 cases were deleted from the sample. Next, CFA was conducted with 282 samples using maximum likelihood estimation. After CFA, three items (i.e., Item1, Item9, and Item11) had standardized parameter estimates smaller than the recommended value of .50 (Hair et al., 2010).
Furthermore, AVE values for factors below .50 were calculated, indicating the absence of convergent validity. After removing these three items, 11 item structure of the PAS was re-subjected to CFA. The diagram regarding the factor structure of programming anxiety with new item codes (See Appendix) and the parameter estimates was given in Figure 2.

Figure 2. Standardized Coefficients for the Two-Factor Model of PAS
According to Hu and Bentler (1999), researchers employ numerous goodness-of-fit metrics to analyze a model. In this study, the Chi-square goodness of fit test, Root Mean Square of Error of Approximation (RMSEA), Standardized Root Mean Square Residuals (SRMR), Goodness of Fit Index (GFI), Normative Fit Index (NFI), and Comparative Fit Index (CFI) were employed to assess model fit (Brown, 2006;Browne & Cudeck, 1993;Hair et al., 2010;Kline, 1998). Table 4 presents the fit statistics for the confirmative factor analysis. When Table 4 was analyzed, the chi-square value (X 2 = 114.226, X 2 / df = 2.79, p = .000) was found to be significant. RMSEA value of .071 indicate good adaptation (Brown, 2006;Browne & Cudeck, 1993). GFI, CFI, and NFI values greater than .90 indicate a good fit (Hair et al., 2010;Kline, 1998). The CFA results indicate that the structural model is a good fit. As shown in Fig. 2., each item loaded significantly on its particular dimension and was relatively large (.50 and above).

Assessment of Reliability and Validity
To assess convergent validity, standardized factor loadings, CR, and AVE were calculated (Hair et al., 2010).
Cronbach alpha values for factors and the whole scale were calculated for reliability analysis (Taber, 2018). The condition that factor loading values greater than .5, Cronbach alpha values greater than .7, and AVE and CR values greater than .5 and .7 were taken into account (Taber, 2018;Hair et al., 2010). Table 5 illustrates that all constructs are reliable since they fulfill the above criteria. Furthermore, each construct's Cronbach's alpha exceeds the recommended value. The Cronbach's alpha value of the whole scale was calculated as .901, which fulfilled the reliability criterion. Furthermore, convergent validity was assessed through (1) comparison of mean values of factors and total programming anxiety depending on gender (Table 6) and (2) correlation analysis of factors, total programming anxiety, and course grade of students (Table 7). As shown in Table 6, female students had more programming anxiety than male respondents. Gender differences in programming anxiety were also investigated using a t-test.
The test results showed that while there was no significant difference between male and female students in self-confidence factor (t = .767, p>.05), there was a significant difference in classmates factor (t =4.058, p=.000) and total programming anxiety (t = 2.518, p=.012). Correlation analysis results (Table 7) revealed that classmates (r=-.194) and self-confidence (r=-.315) factors, as well as total programming anxiety (r=-.284), were negatively associated with course grade, p<.01. All of these correlations were found as weak (Schober, Boer & Schwarte, 2018;Senthilnathan, 2019). The Fornell & Larcker and the HTMT criteria were used to test the scale's discriminant validity. Table 8 summarizes the results of the Fornell & Larcker criteria. In Table 8, each AVE's square root was given on the diagonal, and the correlation coefficients (off-diagonal) for each construct were displayed in the corresponding rows and columns. Fornel and Larcker (1981) state that the AVE values' square root should be higher than the correlations between the components included in the analysis. As shown in Table 8, this condition was satisfied, and the model met the Fornell & Larcker criterion for discriminant validity. In addition to the Fornell & Larcker criterion, discriminant validity was assessed through the HTMT coefficient. The HTMT coefficient was calculated as .705 in this model. According to Henseler, Ringle, and Sarstedt (2015), the HTMT coefficient should be less than .90 if the components to be evaluated are hypothetically close to one other. The HTMT coefficient was found to be below the threshold levels.

Discussion and Conclusion
In computing education research, accurate measurement is critical (Scott, 2015). On the other hand, few measurement tools are available to computer education researchers (Scott & Ghinea, 2014). The current research fills a gap in the literature by developing and validating a measurement tool to assess the computer programming anxiety of university students. The development and validation of the programming anxiety scale in the current study were carried out in harmony with the scale studies recently published in several fields (Nasir, Adil, & Kumar, 2021;Rosario-Hernández, Rovira-Millán, & Blanco-Rovira, 2022;Sun et al., 2022;Zarouali, Boerman, & de Vreese, 2021). EFA, DFA, reliability, and validity analysis resulted in a scale including 11 items, five items for classmates, and six for self-confidence. The minimum obtainable score from the Programming Anxiety Scale is 11, while the maximum score is 55. As the score obtained from the scale increases, programming anxiety also increases. Choo and Cheung's (1991) Computer Programming Anxiety Scale served as a guide to develop the present scale.
In the current study, no factor related to errors was found, while there was a factor for error anxiety in the study of Choo & Cheung (1991) and Demir (2021), in which Choo & Cheung's (1991) scale was adapted into Turkish. Although seven items were included in the draft scale for this factor, six of these items showed high multicollinearity. They were removed from the draft scale due to the exploratory factor analysis, and one item was kept. Although the inclusion of an item about error anxiety shows that debugging is a source of anxiety for students (Dasuki & Quaye, 2016;Nolan & Bergin, 2016), it is worth examining why it is not included as a factor. Not having an "errors" factor may indicate that encountering errors is a source of anxiety regardless of the number of errors and the time spent for debugging, which was utilized as parameters in Choo & Cheung (1991) and Demir (2021). In other words, encountering errors in programs may exist as a single source of anxiety, regardless of the frequency of encountering errors, the number of errors, or the time it takes to debug. The fact that Demir's (2021) study does not contain any information about CFA makes it impossible to make inferences about whether the three-factor model shows a good fit in the Turkish version of the scale and make comparisons about the error factor.
Within the scope of the current study, the only item related to error anxiety was included in the self-confidence factor. One of the reasons for this result may be the differences in perception of debugging as a process in programming activities between the novices and the relatively more experienced individuals. From this point of view, while debugging may be perceived as an independent process for novice programmers, debugging may be perceived as an integral process of programming and an element of self-efficacy perception. Considering a high degree of relationship between self-confidence and self-efficacy perception (Blanco et al., 2020;Malureanu, Panisoara & Lazar, 2021;Tsai, 2019), it is not surprising that an item in the error factor is included in the self-confidence factor. It is evident that enhanced debugging skills develop a programmer's confidence, and fear of making mistakes may be related to programming skills (Ahmadzadeh, Elliman, & Higgins, 2005;Connolly et al., 2009;Nolan & Bergin, 2016). From this point of view, the experience of the participant group of Choo & Cheung's (1991) study on programming was not explained in detail. The participant group was specified only as of grade 12 level. However, participants in both the EFA and CFA stages of the current study were relatively more experienced with programming than the participants in Choo & Cheung's (1991) study. They have developed at least one project and were familiar with the debugging process. This result may indicate that perceptions of debugging are related to programming experience.
Another reason may be the attitudes of the participants towards programming. Choo & Cheung's (1991) participant group consisted of grade 12 junior high school students. In contrast, the current study participants comprised university students who perceived programming as a profession. The students' awareness that the programming profession will include debugging may have caused them to perceive debugging as a personal competence and an aspect of their career. The last reason for this result may be the software development environments (Integrated Development Environments-IDEs) and the resources and materials that can be facilitated for debugging. Scratch was used in Demir (2021) as the program development environment. On the contrary, Visual Studio was used in the current study. The nature of the errors encountered in Scratch and Visual Studio shows differences. The IDE used by Choo & Cheung (1991) was not specified. However, the number of resources found on debugging in the 1990s and today's resources are very different. Today, the internet is used for interpretations of error messages, and previous experiences of other people are utilized for solutions. Nowadays, even IDEs that translate error messages into users' native language exist. Therefore, the characteristics of IDEs and the resources may alter the perception of debugging anxiety.
The validity of the developed scale was tested by comparing the results of studies examining the relationship between programming anxiety and theoretically related variables in the literature. In the current study, female students had more programming anxiety than male respondents. The t-test result also revealed that while there was no significant difference between male and female students in self-confidence, there was a substantial difference in classmates factor and total programming anxiety. This result is consistent with Olipas and Luciano's (2020) study. Similarly, consistent with the former findings ( (Connolly et al., 2007;Figueroa & Amoloza, 2015;Kinnunen & Malmi, 2006;Nolan et al., 2019;Owolabi et al., 2014;Scott, 2015), factors of the PAS and total programming anxiety was correlated with course grades of students. These results indicate that the developed scale is valid and reliable.
Another issue is related to the fact that this scale was developed specifically for programming anxiety. Since this scale was designed specifically to measure programming anxiety, it differs from scales adapted to programming anxiety, such as computer anxiety, computer attitude, and IT anxiety (Chang, 2005;Olipas & Luciano, 2020;Owolabi et al., 2014). There were insufficient instruments to assess programming anxiety in the literature, and the current study offered a psychometrically reliable scale. With the help of the present scale, programming anxiety levels in students can be measured, methods and techniques that can reduce students' anxiety can be developed, and special attention can be paid to students with high programming anxiety. In addition, situations that increase programming anxiety in students can be investigated. The PAS developed in the current study is recommended for study groups that have somewhat experience in creating, coding, and debugging programming projects. Future research may concentrate on a different group of college students from various cultures and countries. Debugging programs is a major worry for me.