The AI Safety Institute Network:

Background

Despite not being an AISI nominally, at the AI Seoul Summit the EU confirmed that the EU AI Office would perform the role of the EU’s AISI. The AI Office was created first and foremost for the purpose of enforcing the EU AI Act.

Although discussions about an EU AI Office date back to the early days of the EU AI Act’s conception (as early as 2020), the AI Office only launched on 21 Feb 2024, 10 months after the UK AISI. On 29 May 2024, the EU announced the leadership and structure of the AI Office, with Lucilla Siolli leading it. This announcement of leadership comes 4 months after the US announcement and a year after the UK’s.

The announcement also leaves two key posts unfilled, the Lead Scientific Advisor and Head of Unit on AI Safety. These three positions will be crucial for the AI Office carrying out its safety-related tasks and fitting into the global network of AISIs.

The AI Office’s initial budget was €46.5 million, reallocated from existing budgets.There are five units in the AI Office, with 140 staff planned to be spread between them. They are:

• Regulation and Compliance Unit
• AI Safety Unit
• Excellence in AI and Robotics Unit
• AI for Societal Good Unit
• AI Innovation and Policy Coordination Unit
• It should be noted that this structure is mostly a repurposing of existing units within the Commission.

These units came into effect on 16 June 2024. Note that AI Safety only makes up one of the Office’s 5 units. How these priorities will be weighted will only be clear once staffing has been fully allocated, but the Regulation and Compliance Unit will necessarily require significant manpower to handle the EU AI Act’s GPAI rules – on top of existing EU digital legislation. With over 86 different EU laws currently affecting the digital sector (and 25 more planned), navigating what decisions are under their jurisdiction will be a key challenge.

While there is, anecdotally, ambition from AI Office’s staff to move quickly and learn from the UK and US AISIs, they may be limited by existing EU rules and procedures. The formal timeline will see the first technical hires only by November 2024, 9 months after launching. There is some uncertainty here, as the Office may also be hiring faster outside of this official channel, via headhunting or direct hires. There is precedent for that. The UK AISI, for example, had both a slower official hiring process, as well as an expedited, more flexible, less official stream.

The two upcoming deadlines for the AI Office are:

1. May 2025 – GPAI Codes of Practice for the AI Act due
2. August 2025 – GPAI provisions of EU AI Act take effect

As of the Seoul AI Safety Summit, there is one additional deadline. At the Summit, participating countries and AI companies agreed to decide on risk thresholds, beyond which companies would not be able to scale or improve their models without first implementing sufficient precautions. The details of those thresholds were left to be decided at the next Summit, on the 10 Feb 2025 in Paris. The clock is now ticking on figuring out where those thresholds should be set. This is an open technical problem that will require technical research and understanding.

The EU AI Office is being set up at the pace to meet its May 2025 deadline. In order to meet the new February 2025 deadline, it may need to move faster.

Vision

The EU AI Office aims to:

1. Support the AI Act and enforce general-purpose AI rules
2. Strengthen the development and use of trustworthy AI
3. Foster international cooperation
4. Cooperate with existing institutions, experts and stakeholders in AI governance

Together with developers and a scientific community, the office will evaluate and test general purpose AI to ensure that AI serves us as humans and uphold our European values Margrethe Vestager, Executive Vice-President for a Europe Fit for the Digital Age

The EU AI Office will evaluate advanced AI models for systemic risks and ensure that they comply with additional requirements set out in the EU AI Act. It will also guide international coordination, including developing codes of practice for compliance with the EU AI Act.

Any general purpose AI (GPAI) models trained on more than 10^25 FLOP are considered to come with “systemic risk” and will have to comply with additional provisions under the AI Act. It’s the EU AI Office’s job to evaluate advanced AI models for systemic risks and ensure that they comply with the additional requirements.

In addition to enforcing the AI Act for advanced AI models, the Office also has the goal of encouraging the adoption of AI technologies across the EU and EU institutions.

“[The EU AI Act] focuses regulation on identifiable risks, provides legal certainty and opens the way for innovation in trustworthy AI.” Ursula Von Der Leyen

Like several AISIs, the EU AI Office also aims to shape international coordination. One example of this are the codes of practice for compliance with the EU AI Act. While industry groups and civil society organisations from around the world are involved in designing them, they ultimately represent an EU effort to promote its vision of international AI governance through a “Brussels effect”. This takes the form of both other jurisdictions directly copying EU codes of practice and the AI Act’s risk framework, or the de facto extraterritorial implications of the AI Act applying to any company that operates in the EU, regardless of origin.

More recently, EU countries aligned with the UK and US to recognize the severe national security risks that advanced AI models could potentially pose. These include capabilities to cause chemical and biological risks, as well as loss of control risks (related to AI deception and autonomous replication).

Background

On 22 May 2024, France announced plans to open a new AI Evaluation Center. This center is a partnership between two established institutions:

1. The French National Institute for Research in Computer Science and Automation (Inria): A public science and technology institute under the joint authority of the Ministry of Higher Education, Research, and Innovation and the Ministry of Economy, Finance, and Recovery.
2. The Laboratory for Metrology and Testing (LNE): The French national agency responsible for designing technical standards, analogous to NIST in the US.

The chief executive of the AI Evaluation Center was expected to start their new position on 1 July 2024. As of early July, no information has been released about who has filled this role, or if it has even been filled yet.

Vision

The AI Evaluation Center aims to become “a national reference body for evaluating all types of AI systems (software solutions or embedded systems, various industrial application sectors).” The Center has three primary objectives:

1. Focus research and innovation on AI Safety
2. Develop new tests and test infrastructures for systems under the EU AI Act’s scope
3. Organise a recurring evaluation campaign of GPAI models

“A profoundly new place, the AI Factory, will host a new AI evaluation center and aims to become one of the world’s largest centers for AI model evaluation, in line with the AI Act.” Emmanuel Macron

Macron aims to have the AI Evaluation Center host a branch of the EU AI Office, to provide the Office technical expertise for model evaluations. Like Singapore’s DTC, the AI Evaluation Center will mostly focus on R&D, while other parts of Europe’s AI Governance structure, such as the AI Office, will handle international coordination and standard setting.

The Center’s creation aligns with Macron’s vision of Paris becoming “the city of artificial intelligence.” However, he has also emphasised a European approach: “Our aim is to Europeanise it [AI], and we’re going to start with a Franco-German initiative.“

“France aims to establish a structure that enables everyone to work together. There is the UK Safety Institute, the US Safety Institute, Korea, and Taiwan. So, it’s not about everyone working on the same subject in parallel. It’s about avoiding wasting time by doing the same thing multiple times. The idea is to divide tasks so that everyone works together concertedly on these subjects. It’s pointless for France, for example, to have different regulations from others.”  Stéphane Jourdain, Head of the Medical Environment Testing Division at LNE, 27 May 2024

Background

The Japanese AISI was announced in December 2023, following the UK AI Safety Summit, and launched in February 2024. On 14 February 2024, the Japan AISI announced its executive leadership, with aims to reach a staff count of several dozen people. However, a target date for reaching this intended staff count has not yet been announced.

In April 2024, the Japanese ministries of Internal Affairs and Communications and the Ministry of Economy released the initial draft of their AI Business Guidelines. This framework was designed to provide advice to businesses and potentially shape future AI regulations. The guidelines were developed based on discussions held at the “AI Network Society Promotion Council” and the “AI Business Operator Guidelines Study Group”, as well as a public consultation for invited comments held in early 2024.

These guidelines include recommendations for AI developers on documentation and transparency, as well as contributing to research on AI quality and reliability and ethical and responsible development guidelines.

Vision

While the exact risks the Japanese AISI will focus on are not yet clear, their mitigation strategies are taking shape. Like the UK AISI, the Japan AISI will primarily focus on three main areas:

1. Safety research (e.g., cybersecurity, AI standards, testing environments)
2. Evaluations
3. International coordination

“You can’t step on the accelerator without a brake. It is dangerous to think only about promoting AI.” Akiko Murakami, Director of Japan AISI

The Japanese AISI’s vision encompasses several key activities:

• Standards collaborations: The Japan AISI has already begun working with NIST and the US AISI on aligning their AI standards.
• Evaluation research: Ongoing research into evaluations, drawing inspiration from similar efforts by Singapore and the UK.
• International coordination: Building upon existing actions, including the Hiroshima AI Process, which kicked off in May 2023 following the Hiroshima G7 Summit. This process resulted in the Hiroshima AI Process Comprehensive Policy Framework in October 2023.
• Information dissemination: Working with both industry and academia to raise awareness of AI safety through seminars and educational material.
• Global collaboration: Participating in various summits, workshops, and roundtables, and engaging in bilateral and multilateral conversations on shared frameworks and standardisation procedures.

Background

On 22 May 2024, Singapore announced that its Digital Trust Centre (DTC) would be transformed into Singapore’s AISI. However, Singapore’s AI Governance efforts preexist this announcement, with the DTC playing an AISI-like role since at least January 2024.

Key milestones in Singapore’s AI governance journey include:

• 23 Jan 2019: Singapore’s Personal Data Protection Commission (PDPD) released the first edition of its Model AI Governance Framework for consultation.
• 21 Jan 2020: PDPD released the 2nd edition of the Model Framework. The PDPD is a sub-agency of the Info-communications Media and Development Authority (IMDA), Singapore’s authority for regulating and developing the infocomm and media sectors as well as fostering innovation, enhancing digital infrastructure and promoting digital literacy and inclusion.
• June 2022: The DTC was founded with a S$50 million (37 million USD) grant from IMDA and Singapore’s National Research Foundation (NRF). It was set up to “lead Singapore’s R&D efforts for trustworthy AI technologies and other trust technologies”, but it is unclear for how long its focus has been on generative AI systems and frontier AI models
• June 2023: IMDA launched the AI Verify Foundation, focusing on AI verification (meeting specifications) and validation (meeting expectations). This foundation brought together stakeholders from various sectors, including industry, academia, and government, to collaborate on creating a robust and trustworthy AI ecosystem. The AI Verify Foundation had a strong focus on building out testing infrastructure. Alongside the AI Verify announcement, IMDA also released a discussion paper analysing distinct risks that could be posed by generative AI.
• January 2024: IMDA and the AI Verify foundation released their Proposed Model AI Governance Framework for Generative AI. This built on the previous Model Framework for traditional AI.
• 6 June 2024: Official Model Framework for Generative AI released.

“To keep pace with advancements in model capabilities, R&D in model safety and alignment needs to be accelerated. Today, the majority of alignment research is conducted by AI companies. The setting up of AI safety R&D institutes or equivalents in UK, US, Japan and Singapore is therefore a positive development that signals commitment to leverage the existing R&D ecosystem as well as invest additional resources (which could include compute and access to models) to drive research for the global good.” Singapore’s Model AI Governance Framework for Generative AI, 6 June 2024

Within this framework, IMDA makes reference to the different AISIs being set up around the world and claims that the DTC is fulfilling a similar role to the AISIs of the UK, US and Japan in conducting “R&D in model safety and alignment”. These claims are then confirmed in the official Model Framework for Generative AI, released on 6 June 2024.

It therefore looks like the DTC has been playing its part within Singapore’s AI Governance structure as its AISI since at least January 2024.

Vision

The Singapore AISI, evolving from the DTC, aims to “address the gaps in global AI Safety Science“. Its focus areas include:

1. Testing and Evaluation
2. Safe Model Design, Development and Deployment
3. Content Assurance/Provenance (Transparency about where and how AI content is generated)
4. Governance and Policy

The Singapore AISI’s vision is rooted in the belief that demonstrating safety can drive innovation and adoption of new technologies in business. As stated in their Model AI Governance Framework for Generative AI:

“Safety and Alignment Research & Development (R&D) – The state-of-the-science today for model safety does not fully cover all risks. Accelerated investment in R&D is required to improve model alignment with human intention and values. Global cooperation among AI safety R&D institutes will be critical to optimise limited resources for maximum impact, and keep pace with commercially driven growth in model capabilities

The DTC was primarily a Research & Development (R&D) institution, so it is expected that the Singapore AISI will retain that focus, matching the efforts of the UK and US AISIs on fundamental AI Safety research and evaluations.

It’s unclear whether the designation as the Singapore AISI will massively change the work the DTC does, beyond providing a clearer focus and allowing it to nicely fit in place with the existing international network of AISIs. As what was essentially a university-based research centre, its ability to engage in international coordination and match some of the other AISI’s domains may be limited, especially prior to being officially designated an AISI. However, these other domains may be covered by the other governance structures and institutions like IMDA and AI Verify.

The Singapore AISI is actively pursuing international & industry collaborations:

• Exploring potential collaborations with the US and UK AISIs on model evaluations.
  On 5 June 2024, the US AISI announced collaboration with the Singapore AISI on “advancing the science of AI Safety”.
• IMDA and AI Verify have partnered with Anthropic for a multilingual red-teaming project to collaborate on “a red-teaming project across four languages (English, Tamil, Mandarin, and Malay) and topics relevant to a Singaporean audience and user base.”

This makes Singapore only the second country to have publicly contributed to the evaluation and red-teaming of frontier AI systems.

Background

The United Kingdom founded one of the first national government institutions focused on advanced AI when it announced its Foundation Model Taskforce (FMT) in April 2023. This taskforce was inspired by the success of the UK Vaccine Taskforce, which accelerated the rollout of Covid-19 vaccines in the country. The Foundation Model Taskforce, later renamed the Frontier AI Taskforce, prioritised speed, talent acquisition, and information gathering.

By June 2023, the Taskforce was launched and had hired its permanent chair, Ian Hogarth.

In November 2023, following the UK-hosted AI Safety Summit, the Frontier AI Taskforce was transformed into the UK AI Safety Institute (UK AISI). This transition built upon the existing structure and priorities of the taskforce.

The UK AISI is centred around three main pillars:

1. Moving Fast: Aiming to keep pace with rapid AI advancements.
2. Hiring Exceptional Talent: Attracting top experts from the AI industry and academia.
3. Focus on Empirical, Measurable Research: lowering uncertainty about AI risks

The initial taskforce format allowed a “startup”-like approach within government – still following existing accountability rules, but being able to operate and hire on much shorter timelines. The UK AISI inherited this startup-like character from its taskforce predecessor.

Political buy-in from the then Prime Minister, Rishi Sunak, enabled the UK AISI to be set up in this manner to prioritise speed.

“In a world of rapidly moving technology, we believe the government can only keep pace if it can ship fast and iterate.” Ian Hogarth, Chair of UK AISI, 20 May 2024

The UK AISI made acquiring talent with experience working in frontier AI companies one of its Key Performance Indicators (KPIs). Eleven weeks after the founding of the taskforce, it had already brought 50 collective years of frontier experience on board, which it expanded to 150 collective years by week 20.

The UK AISI’s initial budget of £110 million helped it recruit top talent. Fostering an institutional culture with somewhat increased flexibility and informality within the traditional civil service structures has also helped attract talent, especially from the existing tech startups and AI companies. In 2014, the Baxendale report listed Culture as one of the primary barriers to attracting talent in the Civil Service. In each of its progress reports, the UK AISI’s Chair has stressed that the fundamental reason the UK AISI exists is to make sure that UK policymakers have access to the best information possible on AI in general, not just on AI Safety. The UK AISI’s leadership hopes that by bringing top private-sector talent into government, the AISI team will advise and enable civil service to dynamically and swiftly draft proportionate and effective policy responses to a constantly changing, rapidly evolving technology. In other words, the ability to acquire and interpret information will allow for greater flexibility.

Vision

The Institute’s current focus areas include:

• Building and conducting evaluations to assess specific potential risks of advanced AI models, and helping others do the same
• Leading international cooperation on AI governance
• Creating the International Scientific Report on the Safety of Advanced AI
• Foundational AI Safety research

“Optimism does not mean blindly ignoring risks.[…] Building the capacity to empirically assess these risks so that policymakers are informed is critical. This is AISI’s core work.” Ian Hogarth, Chair of UK AISI, 20 May 2024

The UK AISI aims to constantly monitor frontier AI systems by running evaluations and experiments on cutting-edge models, while also contributing to base-level AI Safety research. It also prioritises international collaboration, having organised the first AI Safety Summit in November 2023 and co-organizing the second summit with South Korea in May 2024.

The UK AISI focuses on risks from advanced AI models by designing and executing evaluations (tests performed on advanced AI models after and sometimes during their development) to collect empirical data about their ability to pose specific risks. By May 2024, the UK AISI had evaluated 5 state-of-the-art Large Language Models (LLMs) that are accessible to the public, assessing their ability to conduct cyber-attacks, and whether they possess expert-level knowledge in chemistry and biology that could cause harm.

AISI evaluators also examined whether the models could effectively act in an agentic manner, meaning they autonomously pursue set goals and multi-step plans with limited human supervision or intervention. The UK AISI also has a tentative plan to do further research on the evaluation of agents or agentic models, and on broader societal impacts of frontier AI.

The UK AISI is also working on AI safety research beyond just evaluations.

Finally, the UK AISI prioritises international cooperation on AI governance, and plans to foster international partnerships. In line with this goal, the UK AISI organised the first AI Safety Summit in November 2023 in Bletchley Park, UK, leading to the Bletchley declaration. With this first summit, the UK government succeeded in bringing the US and China into the same room to discuss international cooperation on AI governance. The UK AISI then co-organised the second AI Safety Summit with South Korea in May 2024 and commissioned the first International Report on the Safety of Advanced AI, significantly driving and shaping international cooperation on the governance of advanced AI.

The Institute also has the goal of spearheading a network of AISIs around the world with the goal to facilitate international research collaboration and information exchange between AISIs across national borders, with the goal of eliminating the need for AI companies to build relations with AISIs in every country they operate in. So far, the UK AISI has signed international partnerships with the US and Canadian AISIs, and has announced it will open a branch in San Francisco in Summer 2024 to attract and hire more US talent and enable better cooperation with US-based AI companies and the US AISI.

The international partnership with the US AISI paved the way for the UK AISI conducting pre-deployment safety evaluations on Anthropic’s Claude 3.5 Sonnet model and sharing the results with the US AISI.

In April 2024, Nick Clegg, Meta’s president of global affairs, commented that “You can’t have these AI companies jumping through hoops in each and every single different jurisdiction, and from our point of view of course our principal relationship is with the U.S. AI Safety Institute… I think everybody in Silicon Valley is very keen to see whether the U.S. and U.K. institutes work out a way of working together before we work out how to work with them.”

This indicates that the US-UK AISI agreement played an instrumental role in allowing the UK AISI pre-deployment access to evaluate Claude 3.5 Sonnet, given that Anthropic is a US-based company.

Background

The US AISI was announced in November 2023, following the UK AI Safety Summit. Despite being announced just one day before the UK AISI, it only announced its executive leadership on 2 Feb 2024, ten months after the UK AISI announced its own. This delay was due to the US AISI being set up from scratch, unlike the UK AISI which evolved from an existing taskforce.

The US AISI currently has $10 million funding through NIST as part of the 2023/2024 federal budget, which was approved in March 2024. President Biden has proposed a 2024/2025 budget that would allocate $65 million to support the US AISI’s mandate, which has yet to be approved.

These budgets run from 1 October to 30 September, but because of the complex nature of US federal budget approval, often end up being approved halfway through the year.

Vision

“Mitigating the safety risks of a transformative technology is necessary to harness its benefits.” US AISI, 21 May 2024

The US AISI’s vision is “a future where safe AI innovation enables a thriving world”. To get to that future, they lay out a plan to mitigate AI safety risks. The US AISI’s three goals are to:

1. Advance the science of AI safety;
2. Articulate, demonstrate, and disseminate the practices of AI safety; and
3. Support institutions, communities, and coordination around AI safety.

Like the UK AISI, the US Institute’s strategy includes a strong commitment to building out the science of measuring and evaluating the risks of Advanced AI. This science-led approach aims to develop and research best practices and assess risks before legislating.

This is no surprise, since the US AISI is based within the National Institutes of Standards and Technology (NIST), an institution known for advancing the frontiers of “measurement through science, and science through measurement”. In other words, they believe that the best way to learn more and further science is by measuring, and that the process of measurement itself is improved and refined through scientific advancements, in a self-reinforcing cycle.

“AI safety depends on developing the science, but also on the implementation of science-based practices.” US AISI 21 May 2024

The second aspect of the US AISI’s goals is the dissemination of best practices and support of other institutions and communities. The US AISI emphasises collaboration across industry, civil society, and international partners.

Through this collaborative approach, the US AISI has secured agreement from all major AI companies to provide access to their models for pre-deployment testing.

“We have had no pushback, and that’s why I work so closely with these companies,” Gina Raimondo, US Secretary of Commerce, 07 June 2024

More recently, the US AISI signed memorandums of understanding with OpenAI and Anthropic that would give it formal model access both prior to and following the public release of major AI models.

“With these agreements in place, we look forward to beginning our technical collaborations with Anthropic and OpenAI to advance the science of AI safety,” Elizabeth Kelly, director of the U.S. AI Safety Institute.

Building on its agreement with the UK AISI, the U.S. AI Safety Institute plans to “provide feedback to Anthropic and OpenAI on potential safety improvements to their models, in close collaboration with its partners at the U.K. AI Safety Institute”.

The international network of AI Safety Institutes is central to the US AISI’s efforts. It plans to convene international AISI representatives in San Francisco later this year, where it recently opened offices. The US AISI also receives support from the US AISI Consortium, a group of over 200 academic institutions, corporate entities, non-profit and research organisations, and government and policy organisations.

Interestingly, the US AISI frames all of its safety interventions as boosting innovation in the long run.

“Safety breeds trust, trust provides confidence in adoption, and adoption accelerates innovation.” US AISI 21 May 2024

While the UK plans to boost innovation by having its AISI lead to more proportional and effective regulations, the US AISI believes that safety in and of itself is a driving force of innovation, because it increases consumers’ trust in AI technology, increasing adoption, which drives innovation.

A national security lens is another core piece to the US government’s views on AI governance, with concerns about Advanced AI proliferating cyber risks and biorisks. After all, the US’s first major AI policy intervention was restricting the export of high-end AI chips to China, with national security concerns as the sole justification.

Canada and South Korea have announced AISIs, but there is still limited information available on them. Australia, Germany, Italy and Kenya also signed onto the “network of international AISIs”, but have not announced any plans so far.