Digital Security by Design - process and impact evaluation

Challenges

Assessing the impacts against a counterfactual would be challenging. The CHERI technology and the DSbD programme were unique, and without these, the digital security sector would not organically produce a new hardware solution that can enable new secure software design/programming. The number of funded projects was small, and unsuccessful applicants would be unlikely to be able to pursue comparable activities.

The evaluation was therefore based on a non-statistical, theory-based methodology incorporating qualitative and quantitative evidence. It was governed by a “Logic Model” for the programme, showing how the funded activities produce outputs and outcomes which in turn lead to eventual impacts.

Our chosen approach was contribution analysis: assessing the extent to which the activities and inputs of the programme have contributed to the identified outcomes and impacts. This approach was selected because it is useful to understand how projects and businesses function in complex environments where sole attribution of impact is difficult.
The evaluation proceeded through five phases:

1. Evaluation framework development (2000)
2. Baseline report (2020)
3. Process evaluation report (2022)
4. Interim impact evaluation report (2023)
5. Final impact evaluation report (2025)

The programme was due to run from April 2019 to March 2024. The outset of the Covid-19 pandemic necessitated a nine-month delay as much of the key work was laboratory-based. The programme also suffered from global semiconductor supply chain issues at the point of producing the first hardware prototypes, due to Covid-19 impacts and the Russia-Ukraine war.
The programme flexed over the delivery period in response to perceived market needs and availability of additional funding from DCMS and the Defence Science and Technology Laboratory (DSTL) to investigate specific use cases. This entailed amending planned workstreams and delivering new ones:

• Amalgamating separate academic research strands into a single call with a portfolio approach
• Arranging industrial research calls into an initial “de minimis” call for SMEs, followed by a larger-scale industrial research competition, to skill-up potential SME participants in the latter
• Running a joint IUK/EPSRC call to permit both academic- and industry-led bids
• Increasing the number of funded industrial Demonstrators from five to six
• Funding development of RISC-V microcontroller based test boards (the “Sonata Board”) to complement Arm’s prototype design
• Funding for DSTL industrial business partners and a Defence and Security Accelerator competition

The evaluation framework had to be amended to incorporate these changes.

Impacts

The process evaluation, finalised in April 2022, found that the programme was generally being well-run and was making progress towards achieving its eventual objectives. A strength of the programme was that it was able to adapt to emerging challenges and opportunities (for example, reconfiguring the planned competitions to better suit the state of the R&D ecosystem, and seeking alternative supply chains for required hardware).

The process evaluation did identify some potential improvements: delivering written technical reports to the Programme Board, inviting leaders of the largest funded projects to Board meetings, expanding the ambassadorial role of Advisory Board members in industry, and improving monitoring processes for project outcomes and impacts.

The technology platform prototype, known as the “Morello board,” was completed, verified, and delivered for use by funded projects and partners. It has been used to investigate the technology and its potential impact/benefits for new products and services. Technology demonstrator projects were completed in the e-commerce, utilities, automotive, edge computing, and digital computing infrastructure sectors. A “Technology Access Programme” was developed to introduce the technology to SMEs.

At the time of the final impact evaluation, the programme had made good progress in the necessary steps between development of the prototype project and eventual adoption:

• Prioritising the industrial R&D agenda towards cyber security, and memory safety in particular.
• Making progress towards new regulatory standards and legislation proposals on cyber security and memory safety.
• Increasing industrial sectors’ awareness of cyber and digital security issues and market failures, in order to stimulate development of secure products and services and build demand for these.
• Capacity building: formation of new knowledge and skills to develop digitally secure products and services.
• Building confidence in the technology, and demand for the new products and services

Highlights were:

• Programme links with Government through the Advisory Group and appearances in Government strategy documents.
• Building CHERI and the concept of memory safety into policy and regulatory standards, and government procurement; this helps build confidence in the value of the technology, stimulating demand and investment.
• Stimulating collaboration between academia and industry, and building an ecosystem of developers and early adopters.
• Bringing in additional investment from government, and from the private sector, which contributed £238m of co-investment against a target of £117m.
• Stimulating CHERI research outside DSbD: crucially, the Microsoft workstream which was a catalyst for industrial investigation of the technology. This has led to the first commercially available CHERI products being announced using a RISC-V platform.