There were a number of events surrounding the Alliance for CME Annual Meeting. A few of them are worth recapping:
Clarification on Inclusion of Chapter on Engagement with the Environment (Criteria 16-22) in Self Studies — One presenter at the annual meeting quoted from a letter by Dr. Kopelow that implied that all self studies submitted after the March 2010 cohort had to include the chapter on C16-22 regardless of whether the provider intended to qualify for a Level 3 accreditation. This was news to us! In a clarification letter from the ACCME, they indicated they would like providers to explain what they do with regards to the section on Engagement with the Environment, but it was not a mandated requirement. If resources are such that the provider has no intention of qualifying for Level 3 and time doesn’t permit the chapter to be written, it is not required.
Report of an Academic Provider Being Investigated by the Senate Finance Committee Relating to COI — Despite evidence of content review and validation during the planning process for activities developed by a specific academic provider (funded by a major pharmaceutical company) the Senate Finance Committee, aided by an investigative reporter, began an investigation into this multiple intervention activity. All members of the advisory group assembled to oversee and direct the content of the project had COI with the source of funding. The content was also deemed controversial regarding the patient care recommendations included.So the issue of COI and whether it can be resolved by content review was questioned. Regardless of how this turns out, it is clear that the fundamental relationships of key physician content planners must be examined more closely. This issue, I’m afraid, will further play into the hands of those who are advocating that key planners have no promotional relationships with supporters or key manufacturers of products discussed in the activity for at least 12 months prior to the planning process. Due diligence in the area of identification and resolution of COI are imperative!
General Confirmation of Declining Amounts of Commercial Support — It was certainly clear from hallway conversations as well as formal presentations by the pharmaceutical industry, that the overall amount of commercial support over the past year is down significantly from the previous year. Moreover, it also became crystal clear that those organizations that will receive grants going forward must demonstrate (1) a clear understanding of adult learning principles, skills and ability to conduct gap analyses and targeted outcomes measurements that can measure improvement in competence and/or performance and/or improvement in patient outcomes. We can’t emphasize enough how important it is to develop these skill sets as an integral part of every provider’s organization (and not just contract out for them).
On a related note, we have discovered a week-long course on adult learning skills offered by Langevin Learning Services (www.langevin.com) for under $4,000 and another one-day seminar on adult learning. Check it out.
ACCME Explains the Updated Criteria — Steve Singer, from the ACCME staff, offered an intensive symposium at the Alliance that, in our opinion, provided the most detailed information to date on the interpretation of the updated ACCME Criteria for Accreditation. This slide set is worth reviewing and can be accessed as follows: http://www.accme.org/dir_docs/doc_upload/ea6a2e4f-7804-4c11-9dd5-85924f7aa80e_uploaddocument.htm . It is consistent with our trainings and reinforces many of our definitions, such as gap analysis and educational outcomes measurements and their definitions and requirements. It also provides the context for the criteria concerning engagement with the environment.
Everyone knows that the SCS were updated in 2004, providers were required to demonstrate full compliance by May 2005, and yet why are so many organizations having problems with regards to disclosure in their RSS program?
The problem is multifaceted. Here are a few common issues:
Lack of knowledge on who must disclose and when that information is provided to the learners.
How disclosure information is to be made transparent to the learners.
Comprehending the necessity for a reviewer or moderator to provide disclosure information to the provider? To the learners?
Our Findings
In the past, RSS sessions offered by hospitals and medical schools were often not disclosing everyone who had influence over the content of the series. They marched to their own drummer, but in 2005 and again in August 2008 the ACCME made it crystal clear that every activity certified for CME credit must adhere to all of the SCS including RSS.
I often review RSS programs and I find providers are still not fully in compliance with Standards 2.1, 6.1 and 6.2. Everyone knows that the presenters must provide disclosure information to the provider and the provider must implement their mechanisms to identify and resolve COI and then disclose to the learners. But, course directors, CME planning committee members, moderators, and reviewers are often missing from the disclosure information provided to the learners. Why is this I ask? The response I often hear is: “We collect disclosure information from everyone in a position to influence the content and we keep that information in our files.”
This is not in compliance with the Standards of Commercial Support. Everyone—especially the course director and or his/her planning committee—were greatly involved in the development of the RSS. That person or committee had to:
Identify the practice gaps of their own learners
Determine whether or not the activity fit within the “scope of [that department’s] practice”
Assure that the format was appropriate for the setting, the learning objectives, and the desired results (including the rationale for selecting those formats)
Identify universal competencies that would be addressed throughout the series
Draft global objectives, topics, and most likely, select the presenters/faculty.
So, of course they had influence over the content of the CME series.
Like the course director, any moderators who engaged in the presentations, or had any control over the content, also had to make their financial relationships transparent to learners.
If the content was reviewed for resolution of COI, the reviewer also had control or influence over the content, as that person might have made recommendations for changes in the presentation, or asked that additional evidence be cited, etc. Again, this person influenced the content of the session, so any disclosure information provided to the provider (including the fact that there was nothing to disclose) must in turn be communicated to the learners in advance of the session or sessions.
The “Fix”
How can this problem be fixed? There is an easy solution. Develop a template that can be used by all departments offering RSS, as follows:
➢ On the template, designate a “Disclosure” box. In that box, clearly show where “Faculty” information is provided, “Planner” information is provided (this would include the course director and any committee members involved in the development of the series), “Reviewer “(if you use a reviewer, be sure that reviewer does not have any COI related to the session), and “Moderator.”
➢ If you don’t use a moderator, then don’t include that in your template. Make this document real to your organization, so that it will ensure you comply with all of the SCS.
Don’t forget to educate your course directors and department coordinators. If they “don’t know what they don’t know,” they will keep doing what they have been doing for years and years and it will not bear fruit when it is time for you to go through your re-accreditation process. The ACCME expects all CME providers to adhere to all of their Criteria for Accreditation, so RSS need to step-up and demonstrate full compliance as well. By the way, the ACCME is serious about this; we’ve seen clients either go on probation for this or have to submit a progress report!
Once you have educated those in control of the series on how to comply, then you need to keep monitoring to ensure those departmental representatives are making appropriate disclosure to their learners in advance of the activity. If you find they are not, go back and work with them. If they continue to disregard the disclosure requirements, counsel them again. If they still don’t comply, I recommend you adopt a “three strikes” policy. Suspend their credits until they can fully demonstrate compliance.
It’s your accreditation on the line, so work with the departments by educating them, providing them tools to comply and then monitor to ensure disclosure for everyone in a position to control is provided to learners.
I believe that your attitude and approach will garner you the best results. Let your course directors and planners know that you are there to help them, that this is in the best interests of the public and the organization, and obtain support from your management to assure they back you up. Make it a “win-win” and they will comply.
One of the areas of greatest question to our clients has been what constitutes acceptable documentation of compliance with the new Engagement with the Environment Criteria 16‐22. Recently the ACCME issued “Notes and Extracted Examples of Compliance based on November 2008 Provider Performance,” which has provided additional clarity on appropriate documentation for these updated criteria. In the last issue of this column we discussed Criteria 16-20. Now we address the remaining two criteria in this section of the Self Study.
Criterion 21—“The provider participates within an institutional or system framework for quality improvement.”
ACCME NOTE: Evidence of the integration of, and contribution by, the CME provider in healthcare quality improvement.
From Denise: Does the provider (including representatives from the provider) have any direct links to quality improvement activities, either at the institution or through a joint sponsorship relationship? Any links to state or national QI initiatives?
Other Insights from SP&A
This criterion requires that the provider look both internally and externally to show that CME is a tool used to improve quality, that it’s focused on patient outcomes, that is relates to patient safety issues.
Some external relationships to look toward to provide these links are national quality organizations such as the National Committee for Quality Assurance (NCQA) [http://www.ncqa.org], Agency for Healthcare Research and Quality (AHRQ) [http://www.ahrq.gov], the Leapfrog Group for Patient Safety [http://www.leapfroggroup.org],and insurance carriers and risk management companies.
Additional reading: Berwick et al., Connections Between Quality Management and Improvement.Medical Care. 41:1 Suppl; 2003.
If you are a hospital provider, the Leapfrog Group provides comparative quality data on key diagnostic areas as well as “Never Events” for which Medicare and Medicaid will not provide reimbursement, but more importantly for CME those never events provide prima facia evidence of gaps that should be Priority One for your CME Committee.
Criterion 22—The provider is positioned to influence the scope and content of activities/educational interventions.
ACCME NOTE: Evidence of provider’s control of the development of CME activities from inception to idea evaluation.
From Denise: Does the provider control ALL aspects of the CME Program through involvement of staff, planning committees, advisory committees, and Board of Directors? Is the provider strategically positioned to intervene at each step of the CME planning process?
Other Insights from SP&A
This criterion gets at the heart of the new system: that CME is not an afterthought but rather a proactive process where the provider sets the vision and leadership to bring about activities that really improve the health of patients and change learner behaviors.
A performance improvement initiative is a good example of an activity designed to truly change behavior and improve patient outcomes.
While those of us from an adult education background are aware of the importance of using multiple learning methods to appeal to a variety of types of learners, it has only been since the inception of the new criteria that providers have been asked to provide a rationale for their choices. Specifically, C5 asks us to “show the different educational formats (i.e., activity type and methodology) you have utilized for your activities. Explain the rationale or criteria you used in the selection of formats to ensure a format is appropriate for the setting, objectives, and desired results of an activity.” Mere variety, however, doesn’t necessarily mean better activities. The format, as the ACCME suggest, must be suitable for the setting, the objectives and the desired results. It can be as simple as deciding that one format won’t work with a large number of attendees, while other learning formats, though wonderful, may be too time and labor intensive for providers to consider. Also keep in mind that usually the more interactive the activity, the better the evaluation outcomes!
There are also ways to incorporate a few different learning formats within one activity. Choosing to break down a large group into smaller ones, in order to work up and present a case to fellow learners is a great way to incorporate concrete and abstract learners into the same activity. Keep in mind, however, that more careful planning is required. Some questions that should be answered as the planning process unfolds are: How long does it take to give instructions? What instructions are needed? How long should groups work in small groups? How long does it take for each group to present? Who defines individual roles within groups such as the scribe and the presenter? Should you debrief after the small group presentations? How long does that take? Who facilitates the debrief process? What kind of questions do you ask in a debrief?
The point is that while multiple methods make for more meaningful activities, the skills required for designing, implementing, and evaluating are equally as critical to their success, as diversity in the methodology.
To download the Formats Rationale Table, click here. The table represents a number of different learning formats for activities and the rationale for their use.
ACCME
Kolb, D.A. Kolb Learning Style Inventory. Hay Group Transforming Learning; (2007).
Adapted from Effective Adult Learning by Birkenholz
Modifications keep happening to our model CME Planning Document, which we make available to you to use ‘as is’ or modify to fit your planning process. However, remember that CME planning today is, by definition, complex, so simplifying the document is not to your advantage particularly if you are seeking a Level 3 accreditation.
New to this version are the following:
Enhancement for demonstrating compliance with Criterion 3—Designation of intended outcome is tied to each identified gap and a section to notate the rationale for the intended outcome is provided. This last section is a critical feature for the updated ACCME Criteria. What is your rationale for the decisions you made? PEARL: The checkboxes in this section should match what you say in your mission’s purpose statement.
Cross-linked Physician Attributes chart. This allows you to demonstrate multiple competencies from the list that apply to your activity. Conversely, the list should spark ideas for development of content relating to these competencies.
New section on selection of formats—This relates to Criterion 5 and, again, includes a place to record your rationale for the selection of formats designed to engage learners in their own education and reinforce and sustain educational goals.
Outcomes Constructor—If you haven’t seen the planning document for a while, then this section is also new to you. The outcomes builder—based in part on materials supplied by Derek Dietze from ImproveCME—provides options for your planners to use in building outcomes measurements that relate to the designation of intended outcomes from above. Every CME activity today must measure for the outcome you designated in terms of improvements in competence and/or performance and/or patient outcomes.
Sample Case Study—New Appendix C provides a sample case study for use in measuring improvement in competence that was developed for you by Judy Sweetnam.
In January 2008, the ACCME released an updated RSS toolkit. In the memo attached to the toolkit, Dr. Kopelow emphasized the importance of developing RSS activities that support learning and change, as well as improvements in professional practice. He stated that RSS make up 40% of all certified CME in the US and this type of activity must be based on the professional practice gaps of the learners.
The ACCME expanded the requirements for RSS activities in 2008. Monitoring data for each series must continue to be gathered to demonstrate compliance with Criteria 2-11, and providers are now required to list each series within their RSS program on the summary report form that accompanies the Self Study for Accreditation document. Files for series listed on the summary report may be called for review by the ACCME during the accreditation process to demonstrate performance-in-practice.
The ACCME defined its expectation for RSS in terms of meeting Criterion 2 (gap analysis). Dr. Kopelow states, “Professional practice gaps can be those of individuals; however, it is more likely that in an institutional setting the gaps will be those of the healthcare team, or system, in which the learners practice. Providers must deduce the educational need that underlies the professional practice gaps.” He went on to provide questions that will assist providers in assessing the gaps, including the following:
Why is it that the professionals have this gap?
Is it because they do not ‘know’? Is it because they do not have an appropriate strategy in place to address the problem?
Is it that they know what to do, but that they have not, or cannot, implement it?
Accordingly, then, RSS activities must be developed to make a change in clinical competence (strategy), performance or patient outcomes (Criterion 3), and they must be designed to allow for measurement of the changes (Criterion 11) in the series based on the designation outlined in the planning process (e.g., changes in competence and/or performance in practice and/or patient outcomes). Measurement of change can be determined either at the system or healthcare team level, the individual learner level, or within the community of professionals. The ACCME will review the monitoring data to ensure RSS are practice based learning opportunities that impact changes and improvements.
RSS offers opportunities for integrating CME into the process for improving professional practice, development of non-educational strategies to enhance change, collaboration in education, an opportunity to bridge identified barriers to change and to impact the scope and content of education offered within and beyond the organization in which CME is provided. (Criteria 16-22).
SP&A-Recommended Strategies for Demonstrating Your RSS Monitoring System
As mentioned previously, the monitoring requirements have not changed. They are provided as a reminder. Providers that produce RSS are required to:
Implement monitoring systems that demonstrate their RSS meet the ACCME’s Updated Criteria
Provide evidence (e.g., reports) of their monitoring system(s) that meet the following expectations:
The ACCME expects that all series and all sessions within a series will meet ACCME’s Updated Criteria and be in compliance with ACCME Policies. Providers’ monitoring systems must incorporate, measure, and document compliance with Criteria 2 - 11 and applicable ACCME Policies.
The provider must collect data and information from all series as a part of its monitoring system. However, data on each criterion and policy need not be collected from every series. For example, a CME provider may monitor Series A for meeting Criterion 2 and Series B for meeting Criterion.
Monitoring data may be derived from either (1) a sample of a provider’s sessions or (2) from all sessions. However, if sampling is used, it must be applied consistently for 10% to 25% of the sessions within each series across the whole accreditation term.
A provider must analyze the data and information and determine if the RSS has met ACCME’s Criteria 2 - 11 and the applicable ACCME Policies.
A provider must also analyze the data and information for Criteria 16 - 22 (only if the your organization intends to demonstrate it is a Level 3 provider). A provider would indicate that an RSS has met a criterion or is in compliance with an ACCME Policy if its monitoring system indicates performance—as outlined in the criterion or policy—is achieved in 100% of the sample.
The provider will report whether or not it has met Criteria 2-10 and is in compliance with the applicable ACCME Policies within the self study report.
If monitoring system data indicate that performance within the sampled series or sessions did not meet any one of Criterion 2 - 10 or an applicable ACCME Policy, then the provider must:
Identify the problem and describe it within the chapter on Criterion 13
Describe the implemented improvements within the chapter on Criterion 14
Describe the impact of the implemented improvements in the chapter on Criterion 15
One of the areas of greatest question to our clients has been what constitutes acceptable documentation of compliance with the new Engagement with the Environment Criteria 16‐22. Recently the ACCME issued “Notes and Extracted Examples of Compliance based on November, 2008 Provider Performance,” which has provided additional clarity on appropriate documentation for these updated criteria.
Criterion 16:
ACCME Comments
The provider operates in a manner that integrates CME into the process for improving professional practice.
From Denise: Does the educational mission of your organization integrate the CME program into the system-wide focus on improving professional practice? Do your organizational performance improvement initiatives include CME activities? Documentation may include focus groups and/or interviews with learners in your target audience intended to gain a more clear understanding of the nuances of the current practice environment so as to better define a process for improvement; examples of Performance Improvement (PI), Point of Care (POC) or self-directed learning opportunities; any sequential learning reinforcement of CME initiatives.
ACCME NOTE: The provider goes beyond activity planning to show that CME is used as one of the tools to improve the professional practice.
Insights from SP&A
This criterion gets at the heart of the new system: that CME is a tool used to improve quality, that it’s focused on patient outcomes, that is relates to patient safety issues, and that CME is not an afterthought but rather a proactive process.
Get away from responding to available supporter money that spurs CME development; instead, use your CME Committee or Advisory Board to initiate interventions relating to high areas of need. Look at your content areas of excellence and build interventions that address big gaps that are revealed in your research.
Build a mix of multiple interventions that together will enable sustained behavior changes, and use PI and POC activities in your mix.
Criterion 17:
ACCME Comments
The Provider utilizes non‐educational strategies to enhance change as an adjunct to its activities/educational interventions.
From Denise: Consider using qualitative patient focus groups or quantitative surveys for information and
ideas about these types of activities. Examples of supplemental learning tools or mechanisms used to
reinforce desired educational results are: algorithms,patient education tools provided to the physician learner, reminder mechanisms (bulletins on changes or new regulations), office staff support materials, links to websites for additional information.
ACCME NOTE: Evidence of use of strategies such as, but not limited to, rewards, process redesign, peer review, audit feedback, monitoring, reminders.
Insights from SP&A
Almost every activity can use noneducational strategies. Use our most current CME planning document to facilitate this process. If planners don’t suggest using a support tool, press back at them and develop one.
Remember that under Criteria 11‐15, you need to measure the impact of the non-educational intervention used on the outcomes. Be sure to include questions on these tools when you measure IMPROVEMENT IN PERFORMANCE.
Criterion 18:
ACCME Comments
The provider implements educational strategies to remove or address barriers to physician change.
From Denise: Encourage planners to ask about any barriers that learners are likely to encounter what will derail then from achieving the intended result of the activity. Then facilitate a discussion about how the activity can address these obstacles that prevent physicians from making changes in their practice. Notes from course planners, evaluation summaries, ARS responses or outcomes results that demonstrate the barriers are all appropriate forms of additional data.
ACCME NOTE: The provider has data and information that explains patient outcomes, beyond the performance of their learners.
Insights from SP&A
This is an example of thinking outside the box—a key determinant of those providers receiving Level 3 status. Don’t just go through the motions when planning; always be looking at the behavior change that is the hallmark of a CME activity. What might come in the way of learners attaining the results you intend? This is what Criterion 18 addresses.
Again, the latest version of the CME Planning Document facilitates the identification of barriers.
Criterion 19:
ACCME Comments
The provider implements educational strategies to remove, overcome or address barriers to physician change.
From Denise: Tied directly to Criterion 18, providers must include this strategy in the CME program through
activities where course content is designed to remove or overcome identified barriers. Documentation may
include planning documents, planning notes, course materials and mission statement where barriers are identified in content areas.
ACCME NOTE: The provider has data and information on barriers to change applicable to its own learners, and incorporates these insights into its CME program through activities.
Insights from SP&A
This is the second part of the process after the identification of barriers. Barriers to achievement of result can’t be allowed to remain unchallenged. Planners must devise strategies to address each barrier identified above, and your documentation needs to be able to show the content that reflects the strategies to dissolve or remove the barrier.
Criterion 20:
ACCME Comments
The provider builds bridges with other stakeholders through collaboration and cooperation.
From Denise: CME today values sharing ideas and best practices and structuring educational partner
relationships that enhance the learning experience. Documentation may include correspondence with a
collaborator regarding input into development or actual involvement in an activity; planning notes that identify the role a collaborator played in an activity or an activity brochure that identifies collaborators.
ACCME NOTE: The provider allies itself with other organizations in a purposeful manner in efforts to
achieve common interests. These collaborations may support any aspect of the provider’s CME program in service of achieving its mission.
Insights from SP&A
This is more than mere joint sponsorship. It is a “purposeful” strategy to create a team of resources that best supports the activity’s change goals.
When you start to examine ways that your organization can better connect to various specialty boards that enable you to claim ABMS “PIM” and “SAM” status, you will start to see how important these alliances are to achieving those goals.
Currently, neither Criteria 21 nor 22 require attached documentation of provider performance, but must
be discussed thoroughly in the narrative of that chapter of the Self Study. More on Criteria 21 and 22 in
the next edition of The Blog.
Criterion 11 from the ACCME’s Updated Accreditation Criteria (September 2006) states the following: “The provider analyzes changes in learners (competence, performance, or patient outcomes) achieved as a result of the overall program’s activities/educational interventions.”
Several misconceptions regarding ACCME expectations of CME providers with respect to educational outcomes measurement are still in circulation. One of the most common misconceptions is discussed here. Subsequent articles will deal with other misconceptions.
Misconception: CME providers don’t have to measure every CME activity.
Question: “Is an assessment of changes in knowledge, practice or patient outcomes required for each CME activity?
Answer: Yes……at this point an assessment of change is required for each CME activity. The information is then analyzed by the CME provider in the context of the overall program’s effectiveness. Criteria #11 and #12 require the provider to have knowledge of the effectiveness of their Program of CME in the context of changes in competence, performance, or patient outcomes.”
Also, in the recently distributed document, “ACCME’s Accreditation Criteria, Notes and Extracted Examples of Compliance” (November 2008, page 3), relative to Criterion 11, the ACCME states: “Provider seeks to understand the net, or overall, changes in competence, performance, or patient outcomes facilitated by their CME program using data and information from each and every CME activity…” (emphasis added).
However, keep in mind these pearls:
You don’t have to measure every participant.
For Regularly Scheduled Series (RSS), the full set of weekly/monthly meetings are most often collectively considered one CME activity and therefore you wouldn’t necessarily have to conduct outcomes measurement for each meeting in the series (although some providers choose to do this). Some providers use a sampling method to measure an RSS (measure a certain number of meetings during the year).
Likewise, with an annual conference, it is usually considered one large/long CME activity worth many credits, and you wouldn’t have to measure every single presentation that makes up the entire conference.
In general, physician course directors and faculty aren’t up to speed on the new outcomes measurement requirements (or many of the other newer ACCME criteria). You’ll need to spend time explaining the criteria, the important and changing role of course directors and faculty with respect to measurement, and your outcomes measurement strategies. The term “outcomes” means something quite different to a physician than it does to a CME professional.
So the jury is out—CME providers need to measure each and every CME activity to be in compliance with the ACCME’s Updated Accreditation Criteria. The question is, how are you doing on your progress towards using evidence-based methods for measuring, analyzing, and reporting results from each CME activity, and then translating those results into improvements in your CME activities and overall CME program?
Stay tuned for future articles on misperceptions regarding:
Outcomes measurement and Level 3 accreditation status
Measuring changes in knowledge versus competence
The use of objective performance and patient outcomes data versus physician self-reported data.
Alliance for CME Annual Meeting (January 28-31,2009). NOTE: If you want to schedule a brief work session with any member of the SP&A staff at the Alliance meeting, please contact Kathy Cseke as soon as possible to get on the agenda. Kathy can be reached at cseke@passinassociates.com.
March 2009 Cohort Decisions Released (week of March 16)
Upcoming 2009 ACCME Workshops (May 13-15; August 12-13; December 9-10)
July 2009 ACCME Self Study Cohort Decisions Released (week of July 20)
These useful tools are available for downloading and utilization for your CME Program:
Orientation to Planning CME in 2009—A primer for course chairs and other planners based on the CME planning document. This useful tool is a way for you to provide all faculty with a required preparation for planning CME. It lets them know what the requirements are that you, as a provider, must meet and their role in the process. This PowerPoint presentation can be tailored and can be transmitted in an auto-play format or transformed into a video. Download is at www.passinassociates.com/downloads/planningcme2009.ppt.
Preparation Guidance for Taking the CCMEP Exam—This document includes a comprehensive set of readings and guidance to assist you and your staff in preparing to take the next CCMEP exam. It can be downloaded at www.passinassociates.com/downloads/ccmepexamguidance.pdf.
