Office for Students (OfS): Proposed numerical thresholds to underpin requirements for minimum acceptable student outcomes / Universities UK: Framework for programme reviews: Ensuring the value of university courses

Published: 18 January 2022

The OfS has published its long-awaited action plan to tackle “poor quality courses”. It sets out, in a number of technical documents and a consultation, numerical thresholds for course continuation and completion, and professional employment or further study. Thresholds differ depending on the category of the cohort eg full-time first degree, postgraduate taught masters, part-time other degree, apprenticeship etc. It has also published a consultation on changes to the TEF and an analysis of responses to a consultation in November 2020 on the approach of establishing numerical thresholds. The first five points in the “At-a-glance” section below relate to the OfS threshold consultation document.

The OfS documents can be found here.

In anticipation of the OfS announcement, UUK has published a new framework, which is aimed at helping universities in England to identify and improve any course which may “fall short” in quality terms. It is based on work undertaken by an advisory group of vice-chancellors, chaired by Professor Julia Buckingham. A series of roundtables were held in Summer 2021 involving 66 member institutions in England. Separate discussions were held with universities in the devolved nations. Further work on the details of the guidance was undertaken with various HE member organisations and consultation was held with stakeholders from business and third sector organisations. The framework was piloted by a number of institutions in England to understand how it might work, identify barriers and gather evidence on how it can add value to existing processes.

The framework document can be found here. 

At-a-glance:

• New thresholds for continuation range between 60 per cent for first degree part time to 90 per cent for full time postgraduate research. The figure which will apply to full time undergraduate first degree students is 80 per cent (OfS p37)
  
   
• New thresholds for completion range between 55 per cent for part time first degree to 85 per cent for full time PGCE. The completion threshold for full time first degree is 75 per cent (OfS p37)

• Thresholds for progression to professional employment or further study range from 45 per cent for full time other undergraduate to 85 per cent for full time postgraduate research. For full time first degree students, the threshold is 60 per cent (OfS p37)

• Universities and colleges not meeting these thresholds could face investigation, with fines and restrictions on their access to student loan funding available as potential sanctions (OfS p65)

• The OfS is also proposing to create “split indicators” which show, over a four year aggregate, performance within subject areas and across various categories of students, including by personal characteristics such as sex, age, ethnicity and deprivation (OfS p23)

• UUK’s framework recommends that programmes be monitored annually. Where there are ongoing concerns about a programme or course, universities should set out a plan for action with defined measurable milestones, including the transformation or restructuring of courses if needed and, where applicable, closure of a course (UUK p11)

• Universities should publish annual statements showing how courses and programmes are monitored and assessed using core metrics and contextual information on their value. These should be signed off by university councils, or the appropriate executive body (UUK p17)

• The value of courses, including outcomes, should be given equal consideration with other factors related to course viability and financial sustainability (UUK p11)

• Universities should adhere to assurance processes linked to quality and standards (eg UK Quality code for higher education and Professional, Statutory and Regulatory Body requirements). They should also integrate measures and assessment of the quality and value of provision into their annual programme review processes (UUK p11)

• The core themes that metrics should cover are student views, student outcomes and graduates prospects. Universities should integrate at least one or more measures per core theme into their review processes (UUK p13)

• The metrics suggested include student satisfaction, meeting expectations (comparison of outcomes to expectations), continuation, completion and value added, graduate unemployment, highly skilled employment and graduate views of career prospects (UUK p13)

• Contextual themes, including supporting economic growth, social responsibility and university mission and strategy, should also be included in the review process (UUK p15)

• Contextual metrics include employment in high-growth and innovative sectors, high skilled employment in low growth areas, employment or further study in local area, entrepreneurship, social mobility, key attainment gaps, progression into public health and social care professions or teaching, contribution to culture, contributions to the green economy and mission-orientated value (UUK p15)

• Research with participating universities found that all have regular processes to assess the performance of courses. Most use measures such as completion, graduate outcomes, and student satisfaction (via the National Student Survey) as part of their review processes (UUK p20)

• Variation between universities primarily occurs in: the weight placed on measures of value compared with financial sustainability, internal capacity for regular and comprehensive monitoring and the engagement of stakeholders inside and outside the institution (eg students, academic  staff, employers) (UUK p20)

Implications for governance:

The OfS announcement on new regulatory action to tackle so-called “low value” courses has significant implications for governance, not least because universities and colleges not meeting the proposed thresholds for continuation, completion and graduate employment, could face investigation and potential financial sanctions.  

The numerical thresholds, which responses to an initial consultation last year show are generally opposed by the sector, cover full and part-time students at both undergraduate and postgraduate level.

An OfS analysis of full-time first degree students reveals that 36 universities, colleges and other HE providers had continuation outcomes calculated as below the proposed threshold.  About 11,000 students a year (3 per cent) start courses at providers that have continuation outcomes calculated as falling below the threshold proposed.

Earlier data used in the OfS’s proposed cohort tracking measure suggests that around 8,000 students (2 per cent) each year were starting courses at 34 universities that had completion outcomes calculated below the proposed completion threshold.

On progression, 55 providers had outcomes calculated as below the proposed threshold, with around 3,000 students each year (2 per cent) gaining qualifications from institutions with progression outcomes falling below the threshold for professional employment and further study.

Coupled with the new TEF proposal to create a “requires improvement” category, the plan to publish universities’ performance against these thresholds presents a potential risk and could have implications for recruitment, league table standing, reputation and, in the long run, course viability.

According to the OfS, the thresholds were set at a level which “take account of a university or college’s individual circumstances” – a process laid out in a technical document.  However, as the OfS itself points out, students studying on courses below the thresholds are often from groups underrepresented in higher education. If recruitment of disadvantaged students impacts negatively on continuation, completion and professional employment rates, one unintended consequence could be to undermine universities’ incentives to actively seek out such students, stalling the widening participation agenda.

Nicola Dandridge, OfS chief executive, points out in the press release accompanying the consultation that most universities and colleges in England run high quality courses that deliver positive outcomes for students and that the thresholds will not affect them.

However she added: “But we are clear that we are raising our expectations of universities and colleges. Low quality courses which lead to poor outcomes for students are unacceptable, and we are determined to take action where students are recruited onto courses which offer few tangible benefits.”  

UUK’s framework for reviewing the value of courses and programmes was published the day before the OfS regulatory action plan and is set against the backdrop of this “crackdown” on “pockets of low quality provision” 

It recommends that courses and programmes are reviewed annually and should be signed off by university councils or the equivalent. Governors will need to decide what mix of “core” and “contextual” metrics from those suggested by UUK are right for their institution, bearing in mind the new OfS thresholds.

According to the research carried out among universities that participated in the development of the framework, institutions already have regular processes to assess the performance of courses which incorporate measures such as completion, graduate outcomes, and student satisfaction. Reviews mostly occur on an annual basis, with a third of institutions having more extensive reviews every few years, and the same proportion using ongoing monitoring processes.

Governance of processes is also varied, including how decentralised roles and responsibilities are to faculties, schools, and departments.

Historically, graduate outcomes have been used by just over half of institutions in their review processes, with feedback from the pilots suggesting that this has increased markedly in recent years.

Variations in university practices are the result of universities viewing the purpose of reviews differently. Some concentrate on application and recruitment numbers and financial sustainability, for instance. Quality of teaching might be a focus, or finding opportunities for growth, and developing courses.

What the framework makes clear is that going forward, the value of courses, including outcomes, should be given equal consideration to course viability and financial sustainability. In light of this, the framework gives universities and governing bodies an opportunity to reassess the focus and priorities of their current review processes.

According to UUK, putting the framework in place should be relatively low burden, as it is designed to be embedded within existing structures and aligns with wider discussions on quality and value in higher education.

Participating universities saw the merit in including the wider benefits of higher education – particularly social responsibility and economic growth – in programme reviews.

There was strong support for the emphasis placed on contextualising measures in a way that allows programme reviews to be flexible to the nuances of a course’s student base and location. 

There was also a lot of interest in exploring approaches to monitoring social responsibility and contributions to economic growth. While these forms of impact feature in many institutional missions and strategies, they are rarely embedded in programme reviews in a systematic way due to a lack of well-developed good practice in the sector.

Some concerns were raised about data validity. External data sources may only be available across broad subject areas, for instance, which can make drawing conclusions about an individual course difficult. Smaller courses may not have sufficient data points to be used reliably in assessments. Some data sources, particularly those on graduate outcomes, can have a time lag and so are limited in how useful they are for assessing current provision or how they apply to a new cohort or students.

To help to explore and address these issues, UUK will oversee the establishment of a forum on developing quantitative and qualitative data on value.

It will engage with data providers to explore options for introducing centralised, available data for measures in formats that support comparisons within and between institutions.

It will also look at how the use of data, metrics and qualitative contextual information can reflect changing student, graduate, employer, and government views of the value of higher education courses.