Nearly all of the ICER research papers describe an empirical computing education project. (There are a very few exceptions – for more about those, see below). For the typical ICER paper, here are some key factors to include (as an author) and to look for (as a reviewer):
- Are there one or more clearly stated research questions? Since the rest of the paper will be organized around these, it’s often good to put them in the abstract and in the first section of the paper.
- Are the questions of interest to the ICER audience?
- Related work in computing education
- Is the relevant work in computing education included? If not, a good review must give references to missing material. Simply saying “The related work section is incomplete” is not enough.
- Do the authors clearly describe the relationship between the previous work and the current research questions? In what ways does the current project build on the previous work, and how is it different?
- Related work in educational theory
- Is the project based in educational theory?
- Is the theory described clearly, with appropriate citations?
- Is the theory’s relationship to the current project clearly described?
- Is the data gathering sufficiently clearly described that the reader could reproduce it? Some key information to include:
- About the data: why this particular type of data is relevant to your research questions
- About the participants: how many, what was their background (are they instructors, students, alumni, etc.); what if any formal coursework have they had in computing; how many were men and how many women; and any other factors that are relevant to the author’s project
- About the person(s) gathering the data: What is their relationship to the participants? For example, if the data were collected from students in a class, was the instructor one of the researchers or not?
- About the data gathering process: did the project use surveys, interviews, samples of student work, other; If surveys or interviews, exactly what questions were asked; Etc.
- Is the data gathering technique something new to computing education research that might be a contribution in itself?
- Is the data analysis sufficiently clearly described that the reader could reproduce it?
- What methodology was used?
- Is the methodology described, with an appropriate citation?
- Is the implementation of the methodology clearly enough described? How many people were involved? What process was used to resolve any disagreements? Etc.
- Is the analysis methodology something new to computing education research that might be a contribution in itself?
- Are the results of the analysis clearly summarized?
- Are the results thoroughly discussed, including:
- Their relationship to the research questions
- Their relationship to previous work
- Any threats to validity
- The implications of the results for future research
- The implications of the results for teaching
- For model papers from previous ICERs, see the Chairs’ Award papers.
- Non-empirical research papers must be relevant to an audience of empirical computing education researchers and must explain why that is the case. One example of such a paper is K. Sanders et al. “DCER: sharing empirical computer science education data.” ICER-08.