Recently, I conducted a training with a school district in West Virginia. It was for new teachers (1st-3rd Year) in all subjects K-12. There were approximately 75 participants and 20 mentors. One of our activities dealt with Depth of Knowledge. I showed the typical D.O.K., but I wanted them to have a more meaningful experience with D.O.K. levels in mathematics.
The activity I created was inspired by the work of Robert Kaplinsky. I love his Tools to Distinguishing Between Depth of Knowledge Levels. I particularly like his example of sums of whole numbers:
This is a simple and clear example of the D.O.K. progression. However, it does not show D.O.K. Level 4, so I contacted Robert and he directed me to this Problem Post of his, How Many Soda Combos are There on a Coke Freestyle Machine?
Perfect! I compiled these four into one document, scrambling the order, and asked the teachers to discuss the problems in their table groups and to assign a D.O.K. level exclusively to each one. I was intrigued at how different their responses were compared to what Robert (and myself) considered the problems to be. I noted that the group was a broad range of grade levels and subject areas, so I thought I would conduct the same activity with a collection of high school math teachers that I was scheduled to train in California the following week. I was very curious if math teachers would view the problems differently than non-math teachers. Indeed, they did. However, they also disagreed with Robert and me. Below, are the all responses from the groups at each training, as well as Robert’s determination. Notice the variety of responses that was generated within each training.
The choices that earned the most votes looked like this.
Notice that there is not a single example in which all three parties agree. I have no profound analysis of these results; I am simply sharing this very curious experience. I am still pondering the outcomes and their meaning many times over. So much so, that whenever I hear the phrase “D.O.K.,” I smirk and scratch my head.
The Math Projects Journal 
I first learned about Webb’s Depth of Knowledge almost 4 years ago through the Cognitive Rigor Matrix created by Karen Hess. The idea that students could be “creating” without a deep understanding of the concept was a huge a-ha for me. About a year and a half ago I had the opportunity to attend a DoK workshop with Dr. Webb himself that really deepened my understanding of DoK and highlighted some common misconceptions around his work.
Here are my takeaways from that day that I always emphasize when I talk to teachers about this topic that seem relevant here.
#1 – DoK is not about the verbs, it’s all about the definitions of the various levels, and there are different definitions for each content area. It is important to read them carefully and constantly referr back to them. Here they are:
http://facstaff.wcer.wisc.edu/normw/All%20content%20areas%20%20DOK%20levels%2032802.doc
#2 DoK is about complexity, not difficulty.
#3 – DoK is never determined in isolation, but rather through discussion and consensus.
#4 – DoK 1 + DoK 1 + DoK 1 =/= DoK 3 As math teachers, this is really important to understand. We can start with a very rich task and completely eliminate the cognitive demand by “over-scaffolding” for students.
#5 – Higher levels of DoK are not necessarily “better” than lower levels, just different. Students need to encounter work at all levels. DoK 4 tasks may only happen once or twice a school year, and that’s ok.
For those of us who are familiar with the Task Analysis Guide, I’ve found there’s a propensity to equate those levels with the DoK levels. I asked Dr. Webb about this specifically. He said that “Doing Mathematics” tasks are probably DoK 3, but would rarely be DoK 4.
Jennifer, That document that you linked is a gem. Thank you for the extensive comment and the 5-Points from Mrs. Hess & Dr. Webb themselves.