Most everyone who sat for the second annual ACS CCP exam in Madison filled out an exit survey that was sent to them afterward. Their responses will be used to improve the exam process, and to make the entire endeavor even more successful in future years. The survey serves both as a suggestion box as well as a critique sheet.
The Certification Committee has reviewed the exit survey and would like to share some of the feedback with our members. If you took the exam but did not take the survey, and you would like to share your feedback with ACS, please send your comments to Jane Bauer at [email protected].
- One suggestion in the exit survey was that we have a large screen in front of the examination room that illustrates the start-up process and what each candidate should see on their computer screen. It is easier to get up and running if you can see exactly what you should be seeing, rather than being told what you should be seeing. This should expedite the start of the exam, so we do plan to have a screen in Sacramento.
- Some people experienced difficulties using their own laptops for the exam, so a suggestion was made that everyone take the exam on a rental computer. This could be helpful, but we would prefer not to encumber applicants with an additional rental fee on top of the cost of the exam itself. For now, we will leave this option open: you can use your own laptop or you can rent one. What will likely be required in future exams is that those persons using their own laptops will need to arrive earlier than those using rentals—this is to ensure that they are all up and running, ready to go, and to avoid delay.
- One individual commented that a particular question did not apply to their specific circumstance, so they should not have to know that subject. We must restate here that this is a broad-based exam: the cheese industry has many facets that have special relevance to particular jobs; certification is for anyone who meets the eligibility requirements. It would be surprising to see only one question on the exam that did not apply to a candidate’s particular circumstance, and conversely, it would be highly unlikely that any candidate would get a perfect score on the exam.
- One candidate expressed concern that the test was dominated by a company that sent a large number of candidates to take the exam. Companies may enter many candidates for certification, but we must emphasize that no companies sending employees to take the exam have any involvement in the exam writing or exam development. Instead, these companies carefully prepare their applicants for the exam. As companies sending employees to take the exam do NOT have any knowledge of exam questions, and products used by companies to prep their employees for the exam (including flash cards) are not ACS products, ACS cannot guarantee that a company’s prep materials will be relevant to any or all questions on the exam.
- Some applicants ask why they cannot be informed of which specific questions they missed on the exam. This is not standard practice in certifications, and there are several reasons for this. The exam is not completely new each year. Once the exam has been taken, a number of questions fall back into a bank of questions for future years. If any questions were disclosed after the exam, those questions would have to be retired. Assuming that most questions would be missed by at least one test-taker, this means ACS would have to start from scratch and develop an entirely new exam each year. This would require more time from ACS’s volunteer group of question writers and Subject Matter Expects – the individuals who review each question, including the references for the correct answer, and identify “distractors” (incorrect answers) – and it would lead to an increase in exam fees to cover added costs, which would ultimately make certification cost-prohibitive for some. What is provided to the individuals who do not pass is an analysis of how they performed on the exam. For example, if an area of weakness appeared to be sanitation, the person would be advised of this – so they know this is an area deserving more study.
- Another question that came up in the exit survey was why paper and pencil are not permitted in the exam room, for making calculations. Permitting paper during the exam could compromise security; some candidates might attempt to write down questions and “smuggle” their notes out of the room. Attempts to reproduce examination questions by a variety of means are not uncommon, and must be avoided.
- Another frequently asked question is why it takes so long to get the exam results. The process of reviewing the examination’s performance is almost as demanding as the examination construction on the front end. To be as fair as possible to each test-taker, the examination review team analyzes how each item performed against a number of criteria. For example: Did a large percentage of the group choose the wrong answer, and if so, is that wrong answer defensible? Though the exam construction is rigorous and the Subject Matter Experts pour over each item and its answers thoroughly, there is always a chance that something may have been missed. The exam review process also looks at how the group performed as a whole on the examination. If the median score was only 50%, does this suggest that the exam is too difficult? Each item is also assigned a value by each of the Subject Matter Experts independently; is this item something that a CCP should know, or is it something that is nice to know, such as a historical anecdote? The exam review requires careful analysis by our certification company’s psychometrician, the ACS Education & Outreach Manager, Certification Committee members, and Subject Matter Expects. We have discussed ways that this review process can be expedited so that exam takers can receive their results more quickly. Several members of the Certification Committee have offered to forego conference sessions and networking opportunities to work on this on-site. However, to be accomplished successfully, the review process will require more time than all of these parties have available. Our goal is to get the results back to exam takers in a cost-effective way, as quickly as possible, while not compromising the credibility of the certification program itself.
- Some exam-takers expressed frustration with their inability to find cheese educators. To this, I recommend that candidates take advantage of the resources available on the ACS website. Currently it lists 96 cheese educators around North America – and even more importantly, it includes the names of ACS CCP Authorized Educators that offer education aligning with some or all of the Body of Knowledge (BOK) domains that are integral to the exam. The BOK itself was both praised and criticized by exam takers. The BOK is its own entity, somewhat separate from the online directory of educators, but it does include a thorough list of topics that may be found on the exam. It is up to each individual to delve into those areas and to prepare themselves for the exam.
- To the question: In what ways do you believe your business will benefit from achieving certification? The most frequently cited benefits were: better informed customers, resulting in increased sales; enhanced reputation for your business; greater ability to train employees properly; greater customer satisfaction; and increased customer loyalty. This helps explain why more than 90% of the candidates’ expenses for taking the examination were partly or fully subsidized by their employers this year.
Out of a possible 5 points for overall satisfaction with the examination, the average this year was 4.2. Not bad, but clearly there is room for improvement. ACS will be carefully reviewing feedback from the exit survey to make improvements wherever we can, and wherever we feel improvements are warranted. Thank you to everyone who took the survey, as your feedback will benefit exam-takers in future years.
If you are considering applying for the next exam in Sacramento, CA (July 29, 2014), spaces are filling quickly once again. We encourage you to apply as soon as possible: http://www.cheesesociety.org/events-education/certification-2.
We hope to see you in Sacramento!
-The ACS Certification Committee