Shut out of medical school, he blames controversial admissions test which experts say lacks evidence

Erik Soby believes he had the opportunity to attend medical school last year. Torontos scored high on the Standard Medical College Entrance Examination (MCAT) with an impressive average score.

However, most medical schools in Canada now require another admission test (called Casper), and Soby believes Hurdle is his downfall.

“That was an aspect of me below average,” he said. “So I ended up being screened.”

Medical schools are under tremendous pressure every year to make thousands of applications every year – people compete for a coveted place and have the opportunity to become doctors.

To help narrow down candidates, many medical schools use Casper, which represents computer-based assessments to sample individual characteristics.

The company behind the exam, with keen insights, claims Casper helps schools predict which students will achieve career success by evaluating “soft skills” (from empathy, ethics to judgment and communication).

The test raises questions about video and typing scheme-based, requiring applicants to be placed in ethical dilemma.

These issues change every year, but Soby gives an example of appearance.

“They would say, ‘We want to invest in the reputation of this company. [not believing in] He said climate change. You should weigh the two sides of the scene. ”

But Soby said the test was shrouded in mystery—the recipients never gave their actual scores, never understood where they might need improvement, and didn’t know who was evaluating tests that could have such an impact on their future.

Most importantly, critics say the research that Acuity Insight supports its claims is terrible and convincing.

“There is no evidence that Casper can predict future performance,” said Jennifer Cleland, a researcher in the field of choice in the internationally renowned medical school, a professor of medical education studies at Singapore’s Lee Kian Medical College.

Watch | Controversial Casper:

“They are selling this tool, presumably making money from it – people are using it, thinking it is doing what it says.”

Twelve of Canada’s 17 medical schools rely on Casper tests as part of the initial admission process, and many put a lot of emphasis on applicant scores – in some cases up to 30%.

The sharp insight rejected the request for an interview.

A spokesperson wrote that “broad evidence shows that Casper’s effectiveness in evaluating applicants’ non-academic skills” and that the medical school using Casper during his admissions process can identify applicants who stand out not only academically but also academically, but can also act as compassionate and effective physicians. ”

Use casper to spread

The test was developed by McMaster University’s School of Health Sciences and became part of its medical school admission process in 2010.

A few years later, it was licensed by a private company (now Acuity Insights in Toronto) and has received nearly $2.5 million in government funding since 2018 through a grant from the National Research Council of Canada.

Although Casper was originally designed to screen medical school applicants for design, the company has successfully marketed it to other programs in Canada (from nursing, dentistry and physical therapy to undergraduate programs such as the University of Alberta Bachelor of Education Program and the University of Western Ontario Engineering School.

One of the most common criticisms is the lack of transparency in testing.

People who have never told Casper, the company will only send this information to schools.

Instead, testers were told that they belonged to the four levels of the highest to lowest compared to others participating in the test at the same time. Sharp statements make “feedback easier to obtain.”

“I think we should know [the exact percentile]consider how much weight it will bring,” Soby said.

Go Public has heard from more than twenty medical school applicants, and they also have concerns.

“The process is unnecessary,” one wrote, saying he had taken five exams before he finally got admitted to medical school.

“I have endless concerns about the test,” another wrote, saying he wrote Casper three times.

A student who said he took the exam four times wrote that it should be “abolized.”

The company charges applicants $50 for writing tests and an additional $18 submitted to each medical school.

Applicants also questioned the training of those who scored the Casper exam.

Acuity promises the latest online job publishers to earn between $30 and $50 per hour.

The ad does not list any academic or professional requirements and states that “applicants from all walks of life” are welcome and evaluators will receive 65 cents for each written answer they evaluate, for $1 per video response.

Acuity told Go Public that it monitors the speed at which evaluators score “make sure they spend the right time reviewing the context of each response.”

The company also said its evaluators “have different levels of qualification,” which ensures that “they represent the people they will serve when they become physicians.”

“No evidence”

But perhaps the biggest concern is being criticized by respected scholars who say there is no compelling evidence that the test can do so.

After the public announcement of these claims, Acuity sent a long-term document that included eight studies to support Casper.

The Go public shared the studies with Cleland — and three other well-known researchers, with experience in medical school admissions, who refused to be identified because they were afraid of the impact of their majors.

All said the study was weak and insufficient to support the company’s claims.

“I’m actually very surprised at how bad this study is,” Clyland said. “They’re not high-quality research. They’re not very good.”

Cleland and others noted that one of the studies did not examine the actual Casper test, but rather a similar test. They say the other is an overview of existing research and does not provide any new data. Two are conference papers – thus not rigorous in the peer review process – while some do not address long-term results.

“What disappoints me is the lack of scholarships, the lack of strictness, robustness and credibility,” Clyland said.

The researchers also said several studies could be “contradictory” because they were created by co-founders of the company, who are now keen or researchers working for them.

Acuity says “industry-funded research” is a common practice and that all research involving the company has undergone “full disclosure of funds and branches.”

The researchers we spoke with were also concerned that most of the studies were small enough to be scientifically questionable and not replicated.

Cleland noted that a study looked at 31 medical residents and concluded that Casper could predict which professional questions would be less.

“How can you say such a small number?” Clyland asked. “The claim is unfounded.”

Some studies have keen insights suggest that medical school applicants with higher Casper scores are more likely to be invited to interview and do well in that interview, while others have found that Casper can predict who will do well in some assessments in medical school.

But Clyland said no clear model was established.

“So it predicts the performance of a clinical test, but the performance of the second year is not the same,” she said. “You hope that if something is predicting its essential predictions … that would be consistent.”

Few studies track students’ performance over time, but Go examined two keen longitudinal studies in the public.

Someone checked whether Casper’s scores and other admission criteria could predict which medical students might experience professional problems.

“Our study does not reveal a significant relationship between Caper performance and the need for professional remedies in medical schools,” said Lawrence Grierson, senior author of the study, and Lawrence Grierson, associate professor at McMaster University.

Another study conducted by Grierson also found that the Casper test cannot predict who can obtain a medical license on the exam.

“We didn’t find the association,” Gleason said. “It’s hard to know why an association appears in some studies rather than others. However, this means that any statement of the universal validity of the test (at least in predicting future professional behavior) is exaggerated.”

Acuity Insights also put its “technical manual”, a 148-page document that provides a powerful and comprehensive guide to the effectiveness and reliability of Casper tests. The researchers we spoke to refer to the technical manuals not peer-reviewed research, but part of the document for commercial purposes.

The company later sent more research, but some of the studies were duplicate studies they had sent, with several being research papers – rather than peer-reviewed studies – one of which was a student’s PhD dissertation.

Acuity also suggests that this suggests that CASPER tests can increase student diversity, as test results show less racial bias than academic assessments such as MCAT and GPA scores.

Soby wrote the Casper test again last August and is waiting to hear if he will enter medical school in the fall.

Meanwhile, he released Tiktok about his Casper concerns, calling on the fact that applicants don’t get exact scores, and the assessors may be eager to increase their hourly wages by responding.

He said he hopes medical school knows how the Casper test affects people who want to become doctors.

“It’s important to see criticism,” Soby said. “And it’s important that the public sees what’s going on.”