A Critique of Evaluation Methodology Plans

Developing a relevant PICO question that accurately addresses the goal of an evaluation and then locating the most current information on the topic are both key steps in the evaluation process; however, of equal or greater importance is the development of the methodology to gather the data that will answer the PICO question. This is where the evaluator must determine the “who,” “what,” “when,” “where,” and “how” of the evaluation. The evaluation methodology outlines the specific steps that will be taken to complete the evaluation. Who will be involved? What sort of research design should be used? Where is the evaluation taking place? How much time will the evaluation require, and how many participants are needed? How will the evaluation be conducted? It is imperative that the evaluator takes the time to make sure a methodology plan is clear, specific, and thorough.

In this Discussion, you critique a series of poorly constructed evaluation methodology plans, identify areas of weakness, and recommend how they can be improved.

The following scenarios will be used for this Discussion:

Scenario #1: Agnes, the nurse informaticist at a small rural hospital, has been asked to develop an evaluation plan to determine the success of an upcoming training program for the launch of a new computerized nursing documentation system. Agnes has developed the following methodology plan:

“I will speak to participants immediately after the training program to determine the success of the training. They will be asked about the instructor, if the training was a good time length, if there were enough breaks, and if the training location was comfortable. After the implementation, I will ask the physicians and nurses if they like using the new nursing documentation system and how much time it saves them weekly.”

Scenario #2: Maria, a nurse informaticist in a large surgical center, has been asked to develop an evaluation of the implementation of a new Operating Room Management System (ORMS) that includes scheduling, case cart management, and surgical case documentation. Maria has developed the following evaluation methodology plan:

“I will conduct a 30-minute interview with each nurse in the surgical ward to determine his or her impressions of the new ORMS. I will ask them to specify how they log into the system, to detail how often they use it each day, to describe what types of information they utilize, and to provide a detailed list of issues they encounter. I will have the nurses rank 50 different characteristics of the ORMS on a 1 to 100 scale. In addition, I will ask each surgeon to document his or her impressions of the case documentation functions.”

Scenario # 3: The CEO of the hospital system in a major metropolitan area is a brusque, hard-to-please individual. Carl, a newly hired nurse informaticist, has been tasked with developing an evaluation to correspond with the implementation of a health analytic system that the CEO has hand-picked. Carl has developed the following evaluation methodology plan:

“I will arrange one morning where groups of three nurses at a time will have a 15-minute, face-to face meeting with the CEO to both answer his questions and discuss their experiences using the new health analytic system tool. By having this candid dialogue, but without structured questions or parameters, a good overall understanding of the value of the analytic system should be obtained.”

To prepare:

• Review the three evaluation methodology plans outlined within the scenarios above.
• Critique each plan. Is it concrete? Is it specific? What are the strengths? Weaknesses?
• Based on this week’s Learning Resources, recommend at least two changes that would strengthen each plan.
• Research the Walden Library to find an example in the literature of an evaluation study that has a strong evaluation methodology plan, and assess why you believe it to be strong.
• Consider your own PICO question and the elements that would need to be included in the methodology plan to adequately answer this question.

By tomorrow 01/10/2017 before 10pm, write a minimum of 550 words essay in APA format with a minimum of 3 references from the list below, that addresses the level one headings as numbered below:

1)      Post a brief critique of each of the evaluation methodology plans. Describe how each could be strengthened.

2)      Briefly summarize the evaluation study you identified in the Walden Library (include the reference in proper APA format), and explain the elements that made you conclude it has a strong methodology component. (SEE ATTACHED PDF FILES)

3)      Describe how you can utilize what you have observed in both the poor and the strong methodology evaluation plans to ensure that you develop an appropriate methodology to answer your PICO question.

4)      Outline specific elements that would need to be clearly identified in your evaluation methodology, and explain why they are important to include.

Required Readings

    Friedman, C. P., & Wyatt, J. C. (2010). Evaluation methods in biomedical informatics (2nd ed.). New York, NY: Springer Science+Business Media, Inc.

Chapter 4, “The Structure of Objectivist Studies” (pp. 85–112)

 This chapter examines the key concepts in relation to the design of studies and the measurement of results. It includes definitions for fundamental terms, a discussion on the levels of measurement, and a description of the different types of study designs.

  Chapter 9, “Subjectivist Approaches to Evaluation” (pp. 248–266)

This chapter introduces the subjectivist approach to evaluation and highlights the key ways it differs from an objectivist approach. The chapter also examines the premises upon which this type of study is based, and how qualitative data are recorded and analyzed.

    Centers for Disease Control. (n.d.). Evaluation planning: What is it and how do you do it? Retrieved from http://www.cdc.gov/healthcommunication/research/evaluationplanning.pdf

This document provides a brief overview of planning an evaluation, including the different types of evaluations and the components needed in developing the evaluation methodology.

    Stroud, S., & Gansauer, L. (n.d.). Nursing evidence-based nursing practice tool kit: Practice, evidence, and translation process. Spartanburg Regional Health Care System.

This paper provides guidelines for conducting an evaluation. It highlights the different phases of conducting an evaluation and the steps included in each phase.