Do students engaged in reflection between drafts produce better revisions? 
(cf study—Do students who do prewriting/invention before an essay end up with a better final product?)

Research Approach

Quantitative

Knowledge Claims
(methodology)

Post-positivistic

Strategies of Inquiry

Experimental, perhaps quasi-experimental

Two design permutations:
Two treatments—one cohort
Have the same students write one essay where they did not do any reflection. Then have the same students write another similar type of essay where they did do reflection and analyze the differences.

Two treatments—two cohorts
One cohort would write an essay where there was not reflection between the drafts. The other cohort would write with reflections between the drafts—compare the difference. 

The second design would be better because it would control the potential variable of growth and difference between two different writing assignments, but the second would establish a stronger baseline for what revision without reflection is like. The key is whether I could truly say that my samples were equivalent. 

Methods

Content analysis of revisions, statistical analysis of differences

Sample/Sampling

Random sample
The sample would need to be fairly large.  50-100 in each cohort. If significant funding came through, this sample could grow. Again, this sampling could be done at TTU. Large pool of students, doing the same type of essay. One group could be fed reflections; the other group would miss the reflections.  This might have to be done on the first essay to avoid possible influence from previous efforts at reflection. 
How else could I get a random sample?  I could do it like some of the older studies—recruit 50-100 students to come in and write a single essay at one sitting for 2-5 hours and ask them to write a draft of an essay and then revise it.  Without access to a program of writers like TTU, I would have to do it this way.

Collect all drafts and all reflections between drafts.  Analysis of quality of reflections possible ("Irvin reflection scale"). Analysis of both types and quality of changes in revisions but also quality of final product.

Data Analysis

Content analysis of drafts to identify revisions. Holistic scoring of final products to evaluate writing quality.

Possible Results

The test is to see if there is a cause-effect relationship between improved revision and final product and reflection.

Assessment

It is hard to control all variables and isolate reflection as the ONLY determiner of revision quality. But if all conditions are equal/equivalent and the only difference it doing reflection, perhaps it could be a good study.  Sampling looks like a problem, though it could be solved by using the TOPIC vault. 

This study seems very similar to a Clinical type of study, but I guess this study is different in that it sets up two cohorts and compares them. It also seeks a more controlled context. The clinical study might be more of a classroom based case study of eight or twelve students and end up being more descriptive than perhaps? 

WGRA?—experimental studies are mostly about disconfirming, what would be disconfirmed here and who cares? It would disconfirm that reflection has no impact or that reflection does have an impact?

If reflection is framed as an extension of invention (which I plan to do), then classroom teachers used to using invention might be interested, especially if they are seeking more revision help.

Permutation

This design presumes a three draft cycle, so it would involve data analysis of three draft (two revisions). The experiment could be focused on only two drafts and one revision (one reflection).

This type of study could be done with data already in TOPIC if I focused on identifying students who did and did not do reflection. This could allow for a less disruptive type of study.

Mixed Method Possibility
Classroom-based Clinical study with my students—the "treatment" is reflection and I do basically a repeat of the study I did before with better content analysis tools and more rigorous interrater reliability and interview protocols.  This is a qualitative study looking for patterns, one pattern might be tendencies related to revision and text quality. Actually, to match this experimental design, I would have to focus the classroom study on the same question about the relationship between reflection and revision.

TOPIC/Datagogic Triangulation
If I identify certain patterns in my small classroom, can I see the larger patterns identified in a larger pool in TOPIC? Focusing on revision, I could tag all students who did not do any reflections in a three draft cycle and then students who did all three.  Then I could begin to do some datagogic analysis that might in a larger way triangulate—number of changes, change in # of words, final grade (over a large scale, I think we might be able to use the Document Instructors grading as reliable and valid enough to signify something without having to rescore all these essays). 

Another approach might be to find the group of students who might have done no reflections with one essay cycle and with another done all three.  Then compare them datagogically.