SQE2 Business Readiness Review Report

As I noted in my earlier report, setting up a brand new, national, professional exam which is a key constituent part of solicitor qualification is a significant and complex undertaking. It comes after 6 years of consultation; two years of piloting; very significant stakeholder engagement activity and LSB approval. It is inevitable that as the set-up phase is rolled out some features will be improved over time. This report provides an overview of external assurance of the progress to date, by August 2021, with an update in the form of an addendum on some key points in October 2021.

The planning for creation of the first live SQE exams happens against a background of the Covid-19 pandemic. At the time of writing, the plans in place should allow successful delivery of the exams whether social distancing, and / or other public health expectations and restrictions, are needed or not. The planning supports delivery within the current context (October 2021), so, unless Covid-19 related public health restrictions become much more severe, the first SQE1 exams should be delivered effectively in November 2021 and the first SQE2 exams in April 2022.

Indeed, Kaplan have planned for, and have some experience of delivering, scenarios related to Covid-19 issues, such as requirements for ‘bubbles’ and self-isolation of teams. This becomes more important when planning for SQE2 exams because the assessment of some exam elements includes an expectation of face-to-face oral examinations.

SRA have a comprehensive set of delivery expectations, which includes performance targets, delivery standards and contractual requirements. These are being properly managed and are subject to regular and high-quality monitoring and reviews both within Kaplan, and across SRA and Kaplan.

Key observations and recommendations are italicised and summarised at the end of this report.

Methodology and evidence for review

This report aims to offer external assurance that the preparations for the forthcoming first SQE2 exams are on track to deliver high quality, fair, reliable, defensible examinations which will command public confidence and provide a robust and critical pre-requisite for solicitor qualification. This report is a summary of interpreting the evidence currently available. Most of what was reviewed and observed is a work in progress as the detailed preparations for first SQE2 exams in April 2022 continue to develop.

The SQE2 exam comprises 16 assessment ‘stations’ or objective structured clinical examinations (OSCEs). Four are oral exams and the remaining twelve are written exams, taken on a computer, requiring responses which simulate tasks that a Day 1 solicitor would be likely to encounter.

This report provides a snapshot of progress by August 2021, new issues or risks can and will emerge, this report provides a view on status and opinions about readiness.

This review has been completed through access to the following sources of evidence (due to Covid-19 all meetings took place on-line):

• Draft and final first version process documentation
• Draft and final first version policy documentation
• Documents commissioned by Kaplan (produced by Kaplan or specialist third parties commissioned by Kaplan) to assist with their preparations
• High level operational and business continuity plans
• Kaplan’s risk, issues, actions, lessons learned and decisions log
• Management information summary reports used by SRA and Kaplan
• Various documents used to support the commissioning and plan to use new SQE2 venues for oral examinations
• Observation of an on-line training event for examiners preparing for Qualified Lawyer Transfer Scheme (QLTS) currently operated by Kaplan
• Several on-line meetings and one to one interviews and discussions with Kaplan and SRA staff

SQE2 exam task preparation and planning for assessment of SQE2 exams

The draft process SQE2 exam creation documents produced by Kaplan were reviewed. Colleagues at SRA offered review comments on these draft process documents, which I added to. Overall, they represented suitable detail and assurance that the processes, if consistently followed, would lead to the creation of reliable, accurate and fair SQE2 exams.

The Kaplan academic team responsible for the creation of both SQE1 and SQE2 questions and tasks continues to expand as new roles are being placed. New appointments, mainly Subject Heads, have a thorough induction and training programme which includes both the theoretical aspects of what makes a good assessment as well as learning by doing alongside more experienced colleagues. The profile and background of new appointees is aimed to provide significant practical and training experience as well as all being qualified solicitors. Plans to improve the diversity of the Kaplan academic team are developed and these need to be delivered. In Kaplan academic Subject Heads are taking responsibility for specific blueprint areas in which they have appropriate experience and expertise.

The SRA is employing subject matter experts to review and quality assure a sample of draft SQE1 and SQE2 questions/tasks. Kaplan academic teams carry out their own internal assurance activities and checks and external assurance input from SRA subject matter experts should help to achieve the common aim of creating, and then assuring the quality of, assessment tasks. The combined wisdom of these teams should provide an important intellectual asset for the ongoing success of the exam creation process.

Individual SQE2 tasks or assessment stations (also called objective structured clinical examinations (OSCEs)) will include a mix of tasks. Kaplan will use their knowledge and experience of previous tasks to inform the task creation process. Kaplan will have to ensure that tasks are not too similar to tasks previously used. I am satisfied that Kaplan have adequate safeguards in place to make this risk negligible, however the SQE Independent Reviewer and subject matter experts employed by SQE should review this risk after two or three SQE2 exam cycles.

As part of the preparation for the SQE2 oral exams two tasks are assessed by actor assessors. During the SQE2 pilot I observed the training for actor assessors and reflected at the time how professional and high quality this was. These actors are provided by an agency that has experience of working with Kaplan for many years. In discussing potential risks of disruption to SQE2 assessment activities I sought, and received, assurance around contingencies if the current agency used were suddenly not to be available.

Due to venue capacity, there will be two or more ‘runs’ of the SQE2 oral tasks, so that task confidentiality can be maintained for all candidates. In creating several sets of four oral tasks Kaplan will need to pay particular attention to ensuring consistency of demand and difficulty of the tasks. These need to be closely aligned to ensure that whichever set of tasks any candidate takes they are not perceived to be advantaged or disadvantaged because one set is perceived as more or less demanding. While it is important that these tasks are designed to be comparable in demand, the procedures for standard setting will allow adjusting for relative difficulty of individual SQE2 tasks through statistical analysis and professional judgment when the borderline cut-off pass is decided for each task/OSCE. This is an important mitigation if candidates find one or more tasks more or less accessible.

Kaplan have plans to record the on-line assessor training meetings. One of my recommendations from the SQE2 pilot was that such recordings should be made available to the assessors after they have been through the standardisation training to provide a reference point and benchmark to refresh their memories as they proceed through the marking period. I was pleased to be reassured that this is planned in the live context for the first SQE2 exams in April 2022.

Kaplan have good plans in place to train and standardise markers and assessors to be used for each SQE2 exam (OSCE). The addendum to this report summarises the quality assurance activities planned to ensure markers are trained to assess each candidate’s response accurately and consistently.

Kaplan are introducing new software to track in-marking and post-marking progress and performance within and across OSCEs. This will allow real-time review and, if necessary, interventions during the marking period for the SQE2 exams. This offers significant advantages to enable any potential errant marking to be picked up early as well as ensuring that marking is on schedule to meet deadlines.

Overall Kaplan have a good range of qualitative and quantitative measures in place to assure themselves, the SRA and interested stakeholders of the validity and reliability of SQE2 assessments. I recommend Kaplan consider the quality assurance outcomes for a candidate’s overall SQE2 exam outcome when deciding when, and if, to re-mark any individual OSCE task. As planned, Kaplan must be clear who, for each SQE2 task, is the final arbiter of the standard of marking for that assessment task (OSCE).

Candidate information and services

One of the key aspects of preparedness for candidate information and services is the development of: relevant policy and processes; the website and helpdesk functions. I have already reviewed and offered comments on the draft policy and processes and reported on initial progress in my March 2021 report.

The new SQE website is now operational and has accepted registrations and applications for reasonable adjustments for SQE1 exams to be held in November 2021. The functionality and reliability of the new site appears to have met its design aims.

Kaplan has used specialist third parties to provide support to review and research their progress in creating the new website. In addition to functionality reviews in test environments, Kaplan have commissioned ongoing accessibility reports. These reports summarise the simulated use of the test website by users with protected characteristics, with a focus on ensuring all groups, including those with disabilities, can expect the AA access standard planed (which is the internationally recognised standard, or conformance level, many websites are expected to meet eg UK Government public facing websites and is a contractual requirement of SRA on Kaplan). Significant progress has been made since my last report in March 2021 and most amendments required to achieve the AA standard have now been delivered. The remaining changes to attain the AA standard are now scheduled.

An accessibility statement has been published which confirms what the site can support now, and what improvements will be made.

Kaplan’s candidate services helpdesk for SQE is now functioning. As this is a new service and it is supporting a new examination the initial period after launch was supported by an enhanced care reporting process which aimed to quickly highlight any emerging issues, eg concerns about user experience, so that rapid solutions could be found. At the beginning of August 2021, I reviewed the data and statistics which summarised the performance information of the various contact channels available. Early indications are the customer service functions are working very well and there appears to be a very good process in place to review ongoing service levels and ensure that lessons are learned rapidly if any issues arise.

As part of the roll out of candidate services to support the new SQE the provision of reasonable adjustment services for candidates with a disability or condition, to enable fair access to the SQE, is a critical function. By early August 2021 the first applications for reasonable adjustments from candidates were being received and processed ready for the first SQE1 exams to be held in November 2021. Early indications are the process is working well. Kaplan plan to give all candidates that have applied for reasonable adjustments for SQE1 a clear indication of the adjustment they will get for SQE2 and, based on their condition, what further information will be required. For those with a stable condition, Kaplan will provide an unconditional offer meaning the candidate will not have to repeat the application process.

Kaplan are exploring additional new technologies to support candidates with certain conditions which make engaging with a conventional exam on-screen difficult or impossible. It remains important that both SRA and Kaplan continue to research and invest in new technologies to support specific candidate access needs, this needs to be conditional on careful testing of any new software to ensure the candidate experience is positive and such candidates feel supported.

SRA and Kaplan have made positive progress in engaging organisations that represent the interests, or have specialist knowledge, of candidates with particular access needs. There is evidence this is encouraging both organisations to be ambitious in offering relevant support, while being aware of the need to fully test new potential solutions before offering in a live context. At the time of writing the first new software package to support severely sight impaired candidates was in planning to be offered to support the initial November 2021 SQE1 exams.

Kaplan are on plan for the set-up of their Equality and Quality team, who will deliver the process for reasonable adjustments, with initial appointments made, including the ‘Head of’ role, who is a solicitor with considerable knowledge of the Equality Act.

I continue to observe significant and growing engagement by SRA and Kaplan in seeking new solutions to enable fair and equal access to the SQE to candidates with particular access needs.

SQE2 exam logistical planning

Kaplan’s operational team have advanced plans for the delivery of the SQE2 exams, including the booking of the relevant test centres and venues regionally and plans for the creation, formatting and delivery of exams, as well as invigilation. They have plans in place to meet likely capacity demands and have forecast ranges for the likely number of candidates for the first sitting. However, despite forecasting volumes of candidates within a wide range, a risk remains that candidate registrations could outstrip capacity, including venue capacity. Kaplan and SRA need to continue to refine their volume forecasting, using as many relevant sources as possible to aid predictions, to mitigate the risk that candidate entries outstrip the supply of spaces available.

A good deal of the experience that Kaplan have gained by running the QLTS exams is relevant to the operational planning for SQE.

The planning for SQE2 exams (OSCEs) is significantly more logistically complex than SQE1. It requires face-to-face oral examinations as well as exams taken on computers in specially designed test centres. For OSCEs which require oral examination four venues are planned initially, two in London and one in Cardiff and Manchester. The second London location is Kaplan’s own premises and is being purpose built to host oral assessments, the site will be hosting similar QLTS OSCE assessments prior to the first SQE2 sitting in April 2022. The aim is to offer candidates a test site within a reasonable travelling distance for English and Welsh based candidates. Kaplan have undertaken significant research and planning in selecting the venues to meet the exacting requirements of the oral examinations and to ensure that, regardless of venue, candidates have a very similar experience and are subject to the same expectations and standards.

Kaplan have gone to great lengths in the selection process for the new venues in Cardiff and Manchester and the Welsh pilot to be held in September 2021 (see further detail later in this report) will offer an important test of one of these new facilities (Cardiff) and the resourcing of it. This includes trialling video monitoring and Kaplan are using specialist support to conduct a site survey and provide recommendations to enable good functionality while ensuring security and confidentiality of recordings. The trial run in the context of the Welsh pilot provides an important opportunity to learn lessons about operating in new venues ahead of live use in April 2022.

It is important that, planned lessons learned from the trial in Cardiff are acted upon when setting up the new Manchester and London venues. As is planned, the security and confidentiality of the exam and candidates’ personal data will need to be a primary consideration. Essentially the detailed plans in place need to be carefully executed.

In preparation for the SQE2 oral exams Kaplan are planning to employ a third-party printer to print the tasks and materials needed for assessors and candidates to complete each exam. This is a significant and important undertaking given that several runs of 4 tasks are planned, therefore many sets of materials are needed. This printing activity takes place a few weeks ahead of the exam. At the time of writing the selection of the third-party printer to be used had not been finalised.

In finalising the selection of the printing supplier great care should be taken to evaluate the capacity, capability and quality of the printing and, equally important, the security arrangements at the premises. Stringent security protocols are required to prevent the risk of the security of exam materials being breached ahead of the exam. The risks of unauthorised people gaining access to these materials need to be mitigated. Additionally, countermeasures should be planned if the (low) risk of a security breach happening is realised.

I have been satisfied that in the context of current public health advice and restrictions regarding Covid-19 that the SQE2 exams can take place in April 2022. Indeed, Kaplan have recent experience, through their operation of QLTS to run SQE2 type assessments while adhering to current public health advice and guidelines.

Internal quality assurance within Kaplan

A thorough risks, issues, lessons learned, actions and decisions log is being maintained by Kaplan. This document, which is updated and shared regularly with SRA through scheduled governance meetings, forms a key part of Kaplan’s quality assurance activities, of which internal quality assurance is a central part.

Kaplan have newly set up an internal quality assurance function for the SQE. This is headed up by an experienced solicitor and supported by a senior colleague with extensive audit experience. I have met with the relevant head of this section on several occasions this year to fully understand the approach being taken and seek assurance about issues arising.

The framework and model for audit activity, across each of the SQE processes within Kaplan, has a sound and sensible foundation. As would be expected, processes which carry the highest risks feature more heavily in the plan for internal audit activity and these will be audited and reported on more frequently. A good process for involving the head of each function or process prior to audit and then to discuss audit findings with them exists. There is also the provision for unannounced audits. There appears to be a clear and transparent culture to support the sharing of audit findings across the senior leadership team, this culture will be tested when issues arise in the live exam setting. A good start has been made.

I recommend Kaplan continue to invest time, as a senior leadership team, in supporting the right culture around internal quality assurance. The leadership culture appears to support openness and transparency around continuous improvement and learning lessons and discourages a blame culture. This is critical for success in a high stakes’ professional examination context. It is important this is maintained when some operational issues will inevitably arise.

As the outputs from the internal audit function are delivered it is highly likely that changes to processes will need to be made. The principles for updating processes are currently being finalised between Kaplan and SRA.

Equality, diversity and inclusion (EDI)

Kaplan and SRA continue to prioritise the delivery of an EDI strategy which meaningfully addresses the risks and issues of candidates with protected characteristics not having fair and equal access. I reported in my March 2021 report that Kaplan are aware they require further progress to be made during the ongoing set up phase for SQE. This progress includes plans to recruit a more diverse group of Kaplan staff as well as freelancers commissioned to write assessment tasks and act as assessors and markers, including standards setting.

I have seen and heard evidence of significant effort to achieve these outcomes, for example engaging with specialist recruitment agencies and tapping into networks that can access a more diverse group of legal professionals. Kaplan have continued to deliver organisation wide initiatives to support leaders and all colleagues in delivering on their EDI strategy, including during induction and training activities.

Despite these good efforts the delivery of more diverse people into the roles mentioned above is proving difficult to achieve. While detailed plans and initiatives to deliver exist, Kaplan and SRA need to redouble their efforts to reach out to potential sources of expertise, or look to invest in the development of expertise, to deliver their diversity and inclusion goals. This may require further novel and new ideas to achieve this.

Progress on equality issues has been significant and meaningful and I have given examples in the sections above which demonstrate good progress, but the need to keep reviewing what more can be done is ongoing and must not be lost sight of as we enter into the operational delivery phase of SQE.

Update on SQE1 assessment build and standards setting

The first SQE1 exam build progress is very well advanced with the delivery of the exams to the Pearson platform completed, on target, in late August. A very significant number of questions are already in the item bank. Plans for ‘Angoff’ standards setting activities for SQE1 and the set-up of the first Assessment Board are advanced and appear appropriate. Angoff meetings take place before the SQE1 exams are taken by candidates, as planned and required of Angoff judges, all involved in these meetings must understand, and abide by, strict confidentiality protocols to avoid the risk a security breach. Stringent safeguards to protect the integrity of the exam are planned and should be followed through.

Further work to finalise the fine detail of the standards setting Assessment Board process is scheduled. Current plans demonstrate high quality features which are common to what has been evaluated, through academic research, as good practice in similar professional competency-based exam contexts.

Advanced plans for offering a Welsh language pilot examination exist.

This pilot is due to be staged in late September 2021 in Cardiff. Kaplan have made good progress in recruiting Welsh language assessors for the sample exams planned (two written and two oral). Kaplan have sought specialist help to ensure the Welsh language proficiency of potential assessor applicants, all of whom are also solicitors. Good progress has also been made in recruiting a sample group of Welsh language speaking candidates to take the pilot exams.

Kaplan, with support externally, continue to search for further senior assessment expertise where candidates are also solicitors (or legal experts) and highly proficient users of Welsh language. Such skills sets are very scarce, and it is possible that the combination of Welsh language speaking assessors and highly qualified Welsh translators will need to be the fallback position adopted. Additionally, all Welsh language assessors will also mark English speaking candidate scripts (when marking identical assessment tasks), which will help to ensure standards across either language offering are strictly equivalent.

In addition to enabling plans for offering a Welsh language version of the SQE, the pilot has the added benefit of allowing a new examination venue, which will be used for all SQE2 oral assessment windows, to be tested in pilot condition, ahead of first live use in April 2022. This is reassuring given the complexity of hosting SQE2 exams at different venues and will enable all aspects of the exam environment, such as: Covid-19 protocols; use of CCTV to record oral exams and staff using different venues, to be properly tested.

In creating several sets of four oral tasks Kaplan will need to pay particular attention to ensuring consistency of demand and difficulty of the tasks. These need to be closely aligned to ensure that whichever set of four tasks any candidate takes they are not perceived to be advantaged or disadvantaged because one set is perceived to be more or less demanding.

I recommend Kaplan consider the quality assurance outcomes for a candidate’s overall SQE2 exam outcome when deciding when, and if, to remark any individual OSCE task. As planned, Kaplan must be clear who, for each SQE2 task, is the final arbiter of the standard of marking for that assessment (OSCE).

Angoff meetings take place before the SQE1 exams are taken by candidates, as planned and required of Angoff judges, all involved in these meetings must understand, and abide by, strict confidentiality protocols to avoid the risk of a security breach. Stringent safeguards to protect the integrity of the exam are planned and should be followed through.

It remains important that both SRA and Kaplan continue to research and invest in new technologies to support specific candidate access needs, this needs to be conditional on careful testing of any new software to ensure the candidate experience is positive and such candidates feel supported.

Kaplan and SRA need to continue to refine their volume forecasting, using as many relevant sources as possible to aid predictions, to mitigate the risk that candidate entries outstrip the supply of spaces available.

It is important that the planned lessons learned from the trial in Cardiff are acted upon when setting up the new Manchester and London venues. As is planned, the security and confidentiality of the exam and candidates’ personal data will need to be a primary consideration. Essentially the detailed plans in place need to be carefully executed.

In finalising the selection of the printing supplier great care should be taken to evaluate the capacity, capability and quality of the printing and, equally important, the security arrangements at the premises. Stringent security protocols are required to prevent the risk of the security of exam materials being breached ahead of the exam. The risks of unauthorised people gaining access to these materials need to be mitigated. Additionally, countermeasures should be planned if the (low) risk of a security breach happening is realised.

I recommend Kaplan continue to invest time, as a senior leadership team, in supporting the right culture around internal quality assurance. The leadership culture appears to support openness and transparency around continuous improvement and learning lessons and discourages a blame culture. This is critical for success in a high stakes’ professional examination context. It is important this is maintained as and when some operational issues will inevitably arise.

While detailed plans and initiatives to deliver exist, Kaplan and SRA need to redouble their efforts to reach out to potential sources of expertise, or look to invest in the development of expertise, to deliver their diversity and inclusion goals. This may require further novel and new ideas to achieve this.

In October 2021 I returned to some key issues related to the set-up of the SQE2 exams, specifically: quality assurance of SQE2 ‘written’ task marking, outcomes of the Welsh pilot held in September 2021 and a review of key risks. I also received an update on the final planning stages for the first SQE1 exams to be held in November 2021. I provide a brief update on these points.

The SQE2 written tasks, which comprise 12 of the 16 SQE2 OSCEs, are marked by solicitor assessors. I have reviewed the extensive quality assurance plans that are being planned to ensure accurate and consistent assessment takes place. Specifically, this involves:

• learning from best practice methods adopted in equivalent high stakes assessment contexts and consulting assessment experts
• high quality training events for all markers on each OSCE
• being clear on who is the final arbiter for the marking standard for each written task
• markers required to reference their live marking against the standard agreed at the training events
• opportunities for markers to seek support and guidance from colleagues and senior markers during live marking
• approximately 10% of provisionally marked scripts to be reviewed by a senior marker
• ensuring markers receive work to mark which included those candidates resitting and taking the exam for the first time
• use of software to track marking trends and flag any unusual marking patterns for review and take appropriate action.

I am satisfied that the package of measures in place should enable a high degree of confidence in the accuracy and consistency of the marking of SQE2 written tasks.

The Welsh pilot was conducted as planned in September 2021. The pilot was very successful, not just in demonstrating capability to provide Welsh language SQE assessment for Welsh language speaking candidates but also in providing learning to apply in the live context, including feedback received from the pilot candidates involved. In addition to providing a test of the provision of Welsh language assessment, the pilot also acted as a trial of a SQE2 regional venue which will be used for oral SQE2 assessment windows. Once more, overall the operational processes worked well, including video recording of the oral pilot exams, and valuable lessons were learned which will supplement detailed protocols planned e.g. operating in a more ‘public’ hotel venue.

In reviewing the key operational risks for the first sitting of the SQE2 exams a primary consideration remains maximising capacity while carefully ensuring all candidates receive a consistent, fair and high-quality experience. It is important that Kaplan plan to maximise the number of candidates that can sit the early sessions of the SQE2, it is equally important to balance the desire to increase spaces available alongside managing the risks of fairness and consistency of experience for all candidates. Kaplan have some good plans to increase capacity over time and are conscious of the potential ‘front-loaded’ demand for spaces which will occur as candidates fail to achieve the QLTS before it is discontinued. The risks in this space will need careful review after the first live sittings of the SQE2.

I received an update on the very advanced plans for the first sitting of SQE1 to be held in November 2021. Current preparations appear to be running to plan and preparatory activities such as the Angoff panels to review SQE1 questions have progressed well.