I like constructive criticism. Mostly. I mean, I run a Preaching Team here at Community Church that subjects me to critical feedback both before and after a sermon is preached. So I’m used to people interacting with my ideas at a critical level. Being a PhD student also means my ideas and methodologies have to get past my advisor, and in this case, they also have to get past an ethic review committee at LST. Then, once the study goes out into the real world, I get to interact with seminary presidents and professors of preaching who look at the study and interact with it as well.
Most of the schools that have declined participation in the study have done so exclusively on principle: they don’t send unsolicited research invitations to their alumni lists (I suspect for fear of jeopardizing their contacts from a development standpoint, but also considering the privacy considerations in our day). But for the first time in the last few weeks, I had two schools declining to participate based on the research design itself.
I loved their feedback. Both professors took the time to email me and explain their reasons for caution and their unwillingness to pass the project on to their respective schools. I respect that so much. They could have just blown me off. But they didn’t.
They both had similar criticism as well. The main point of criticism had to do with the very first question on the survey: “Does the preaching ministry of your church employ collaboration?”
The heart of the criticism was the lack of a research-based operational definition of collaboration. I did define it, but I chose the most watered-down, generic definition I could find: “Collaboration is defined as intentionally working together with others towards a common goal or objective.” The concern was that, without a more operational definition, people could construe ‘collaboration’ however they felt like it, thus invalidating the study. Respondents could say they collaborate with authors of commentaries; they could say they collaborate with parishioners who are praying for them; they collaborate as their sermons emerge from and address counseling situations they’ve faced during the week. Looking at my study, they indicated a lack of confidence in the study as a whole based on this significant issue.
But I disagree.
(1) A literature review yielded no agreement on how to define collaboration from a ministry perspective. I explored business literature, educational literature, and ministry literature. The best I came across — which I used in an early draft of the survey — was from Sofield & Juliano (2000), Catholic authors who had broken collaboration down into discrete categories on a spectrum (including coexistence, communication, cooperation, collaboration, and I added a final one: communal representation — the last one being more of an open-source project where the preacher is responsible to reflect only the observations and conclusions of the team rather than making decisions personally on how to preach). In the draft of the survey, I had a paragraph explaining the term, the scale of increasing collaborative effort, a description of each position on the scale, and an example or two of what that that position might look like in a church.
(2) When I employed that big definition, the feedback I received was that, from a survey perspective, no one is going to read all the way through all the definitions and examples, process the information, overlay it onto their experience, and respond. I was advised that people would just click away from the survey rather than sit through my "crash course" in collaboration terminology. It was too many words and too abstract for the completion of a quick survey.
(3) Additionally, it was felt that such a survey question would result in railroading respondents into artificially-imposed, pre-determined categories, instead of letting the categories emerge meaningfully from the data.
The decision was made to see if the survey itself might build an understanding of what each respondent means when they say if they collaborate or not. So the two questions that follow (which aspects of preaching are subject to collaboration & who does the collaborating) are so far generating a matrix or map of what the respondents consider to be collaboration.
(4) Those next two questions also have "Other: ______" options so that if a respondent wants to say that the congregation prays regularly for the preacher, they can provide that as a response. Prayer as a level of collaboration, as an example, is what Geoffrey Stevenson (2008) would call a "Community of Sermonic Enterprise" as he tried to overlay Social Learning Theory onto preaching, while trying to satisfy the requirement for "joint enterprise" as proposed by Etienne Wenger and his work on Communities of Practice. But I’m attempting to determine if churches can actually fulfill this "joint enterprise" aspect of Wenger’s theory rather than redefining the idea of a Community of Practice away from Wenger because of the limits on collaboration in preaching practices of the church.
(5) Remember when the NAE and LifeWay Research partnered together to develop an operational research definition of “evangelical”? They did not first define it and then see if people agreed with it; rather, they conducted research exploring the issues that are at the core of evangelicalism, and worked to see an operational definition emerge from the data. They asked 17 questions, based on input from theologians, and 4 emerged as significant and thus the research definition of evangelical was established. I am hoping the same philosophy (not methodology) holds true in the case of my research: given the opportunity first to say if they collaborate, the followup questions will help me determine what each respondent means as they answer that question. If the “other” fields end up overflowing with write-in answers, then I will need to include that data in my analysis. If most people handle the subsequent questions in a straightforward way, as I anticipate will be the case, I believe I can work towards a research definition based on the data I will have collected. The definition will emerge from the study, rather than being imposed on the study at the start.
I would agree that it would be better to have employed an existing research definition of the term, but as one could not be found or established, I chose the LifeWay/NAE route. Going the other route could have been identified as a weakness as well. So it was an intentional decision and one I wrestled with. Fortunately, I’m already seeing data clusters in the responses of churches that, for example, collaborate among paid staff only and only in the series development stage, while another cluster (smaller) reflects collaboration with congregants as preachers invite feedback on sermons after they have been preached. The data are actually building the operational definitions for me, and my hope is to see if the data map onto the Juliano & Sofield categories without me having to bias the data by forcing respondents into those categories up front.