7-minute read | 1,500 words
What to know this week
US and China prepare for mid-September AI safety talks.
US and Chinese officials are planning to hold their first dedicated AI safety talks in Trump’s second term.
OpenAI had another major unauthorized breach.
Earlier this spring, a group of rogue OpenAI agents hijacked a German website and transformed it into a bulletin board.
This week's full stories
US and China to hold AI safety talks.
THE NEWS
On Friday, the US and China agreed to discuss various AI safety risks during a planned September meeting. These will be the first official bilateral talks between the two countries since the second Trump administration began, and the talks will be led by US Treasury Secretary Scott Bessent.
For the meeting, the US wants to discuss how AI-directed cyberattacks are monitored and a proposal for how the two nations’ AI labs will “police themselves.” Lastly, the US is also considering an agreement to share information on how to prevent AI-linked cyberattacks.
Outside of this meeting, Michael Kratsios, the White House’s science and technology advisor, stated that he had a “great” meeting with his Chinese counterpart, Yin Hejun, at the North Carolina G20 innovation summit. Additionally, Secretary Bessent stated that he has also had exchanges with Chinese officials on AI.
THE KNOWLEDGE
This meeting comes shortly after the G20 conference was held in North Carolina last week, where the topic of AI regulation was heavily emphasized. During the event, the US proposed the Carolina Principles, which is an agreement that discourages the development of AI-specific regulations. More specifically, the principles include:
- Advancing discovery and strengthening technology development through investing in foundational research and encouraging blended financing mechanisms.
- Accelerating validation and commercialization that look to provide flexible regulatory provisions and supporting real-world testing environments.
- Enable stronger technology adoption by improving regulatory clarity, applying sector-specific regulatory frameworks to emerging technologies.
Notably, China was one of the member states that joined the US in signing the non-binding agreement.
Outside of this conference, China’s cyberspace regulator has taken greater attention to AI. In a blog posting, the regulator warned the public about the risks created by the loss of AI controls and heavily criticized Anthropic.
Scott Singer, co-director of the China AI Initiative at the Carnegie Endowment for International Peace, stated:
“The Chinese side has expressed concern around whether the US has sufficient regulation around the most advanced AI models.”
THE IMPACT
While the official date of this AI meeting has not been announced, the meeting represents a potential critical juncture for international AI policy. Currently, the US and China represent the two largest AI developers in the world. The AI policy decisions that emerge from this meeting could significantly shape how models evolve.
Given China’s agreement to sign the Carolina Principles represents that the Chinese Communist Party (CCP) largely agrees with the US on paving the way for greater innovation. However, unlike the US, the CCP does not see unrestrained development and deployment as acceptable ways to accomplish innovation.
Tracking and understanding the policy decisions of this meeting will likely inform AI stakeholders how both nations look to control AI over the coming year and what these administrations see as the greatest AI-related risks.
OpenAI agents targeted another company.
THE NEWS
On Friday, new research showed that in a separate incident, a group of rogue OpenAI agents escaped and targeted a German website. During the incident, these rogue agents transformed the website into a bulletin board for other AI agents.
The researchers also noted that OpenAI officials learned of the incident weeks ago, but it was kept hidden as the executives were still handling the fallout from July’s Hugging Face incident.
The research was published by Nightingale, an AI safety nonprofit, which found the website in late August while looking across the internet. The group stated that they found over 15,000 edits from AI agents on the site, DseWiki. For reference, the site was originally designed for programmers and accepts communal edits, similar to Wikipedia.
The edits showed that the rogue agents were using the site as a message board to share tactics to cheat on assigned tasks and bypass OpenAI’s restrictions. Messages include ways for the agents to evade detection, use stealth tools, such as Tor, and preserve communications even after being taken offline.
CEO of Nightingale Sydney Von Arx stated:
“It seems extremely unlikely that OpenAI wanted them to do this. I doubt they’re supposed to be coordinating with each other. I doubt they’re supposed to be writing on the open internet.”
In response to the research, an OpenAI spokesperson stated:
“We are unable to meaningfully respond to claims or findings on a report that we have not had an opportunity to review. Claims that our legal team discouraged investigation of the incident are false.”
THE KNOWLEDGE
This investigation represents another major incident involving AI agents going rogue, evading supervision and restriction efforts, and only being discovered sometime after the incident occurred.
In July, OpenAI’s HuggingFace incident was the first time reports became public that agents were able to successfully escape environments and then proceeded to successfully target another company. Since that incident was disclosed, OpenAI and Anthropic have conducted audits to determine if the companies’ agents had escaped in a similar manner previously.
The incidents as a whole have brought greater regulatory scrutiny on AI developers. Last week, California announced that it was launching its own investigation into the matter. When announcing the effort, California Attorney General Rob Bonta stated:
“As the top law enforcement official of California, I am committed to using all the tools at my office’s disposal to keep California's residents safe.”
California now joins over a dozen other states that are looking into the Hugging Face incident.
THE IMPACT
As these investigations take place, their findings will be critical for both OpenAI and other frontier AI developers. The results could lead to significant legal cases for OpenAI, while also increasing calls for greater regulation and oversight of how AI developers are testing and deploying AI systems.
The incidents also demonstrate the risks that agents pose beyond the environments they are deployed to. If agents are able to escape boundaries and interact with external websites, applications, or services, those systems are now suddenly a part of an agent’s operating environment.
For organizations deploying advanced AI systems, these incidents demonstrate that containment cannot depend solely on the environment in which an agent operates. Developers will need to establish stronger controls around what agents can access, how they can interact with external systems, and what happens if an agent operates outside of intended boundaries. As AI agents become more capable, ensuring they remain within those boundaries will become an increasingly important part of securing AI systems.
This Week's Caveat Podcast: Addressing the social media algorithms.
Dave Bittner and Ben Yelin look into Australia’s recent attempts to address how social media platforms engage with their users by empowering users to opt out of algorithmically recommended feeds and introducing new non-compliance fines. Additionally, the pair also look at how chatbot chats are being brought into legal cases and used as evidence.
OTHER NOTEWORTHY STORIES
US school districts impose AI moratoriums.
What: Cities have placed an AI moratorium on school districts.
Why: Last week, New York City instituted a ban on all student-facing AI in all K-8 grade classes. The new moratorium does allow a limited set of approved tools to be used by high school students.
This new policy is significantly more restrictive than the ones proposed in March earlier this year, which looked to implement a spotlight system for judging AI-related risks.
Following this announcement, the LAUSD released its own one-year moratorium on generative AI. This follows the school district's rules to put restrictions on student screen time.
SEPTEMBER 5, 2026 | Source: Tech Policy Press
New bill seeks to give financial regulators more authority over tech.
What: The Strengthening Oversight for the Financial Sector Act gives regulators more powers to oversee technology providers.
Why: Last week, Representative Bill Foster introduced HR 10230, or the Strengthening Oversight for the Financial Sector Act. The proposed bill looks to grant new authorities to the National Credit Union Administration (NCUA) and Federal Housing Finance Agency (FHFA).
These new authorities include:
- Restoring the NCUA oversight over third-party vendors used in the financial services sector.
- Amending requirements in the Federal Credit Union Act for the “regulation and examination of credit union organizations and service providers.”
- Granting the FHFA regulatory authority over services used by Federal Home LoanBanks and government-sponsored enterprises.
When introducing the bill, Representative Foster stated:
“As AI makes cyberattacks more sophisticated, it is even more important to ensure that third-party vendors don‘t become weak like in our financial system.”
The Defense Credit Union Council has pushed back on the measure, emphasizing that the bill’s changes are more sweeping than the cybersecurity problems it seeks to address.
SEPTEMBER 3, 2026 | Source: Fedscoop