Glad to have you back at TechCrunch Mobility, your destination for advancements in transportation and the increasingly significant role of AI within it. To receive updates directly to your inbox, sign up here for free — simply click TechCrunch Mobility!

In a few days, the Amazon-owned Zoox will commence charging for robotaxi services. While this may not appear significant since the firm’s custom-designed robotaxis are currently transporting passengers in Las Vegas and San Francisco, it is also launching an early rider initiative in Miami and Austin. 

However, none of this is economically relevant until the company achieves commercial operation. 

As of August 10, it can do so, thanks to an exemption granted by the National Highway Traffic Safety Administration (NHTSA). Because Zoox vehicles do not have several conventional controls required by federal law, such as a steering wheel and pedals, it required an exemption from federal motor vehicle standards for operation. An existing exemption permitted technology demonstrations; the new one enables Zoox to operate a commercial fleet of up to 2,500 vehicles for two years. 

This exemption signifies a victory for Zoox, but it also opens avenues for other autonomous vehicle developers aiming to introduce robotaxis that don’t conform to traditional requirements like steering wheels, pedals, or other features deemed unnecessary when there’s no human driver. For example, a robotaxi effectively utilizes sensors on its exterior to provide visibility rather than needing a rearview mirror. 

Tesla stands to gain significantly as it develops its two-seater Cybercab, but there will certainly be others benefiting as well.

---

In other updates, you might have missed my thorough evaluation of Uber’s autonomous vehicle enterprise. Take a look; it provides an exhaustive overview of every company Uber has partnered with (and invested in at times). More partnerships are anticipated. 

You might remember that the Financial Times conducted its own analysis of Uber’s investments in AVs, concluding it was around $10 billion. Interestingly, Uber CEO Dara Khosrowshahi confirmed this figure during the company’s earnings call, stating that the company plans to invest $10 billion “over the coming years” to roll out 120,000 driverless vehicles.

Deals!

Moove began as an African fintech company that provided vehicle financing solutions for app-based drivers. While Moove still operates within the mobility sector, it has transformed into a major ride-hail fleet operator (42,000 vehicles and expanding) across 13 countries. 

It has also established a new division concentrating on autonomous vehicles. Moove achieved an early victory by becoming the fleet operator for Waymo in Phoenix, Miami, Las Vegas, and, in the future, London. However, co-CEO Ladi Delano clearly has larger aspirations for Moove — and he now possesses the fresh investment to pursue those goals. 

Now based in Dubai, the startup secured $250 million in a Series C funding round led by Mubadala Investment Company with Woven Capital and Ion Pacific acting as co-leads. Moove, valued at $2.1 billion, intends to use the funds to enhance its management of autonomous vehicle fleets, which includes hiring around 350 employees. 

Notably, Moove plans to acquire Waymo robotaxis while already owning robotaxi assets from another undisclosed entity.

Other deals that caught my attention this week …

Accell Group Holding, the Dutch bicycle manufacturer behind brands like Lapierre, Raleigh, and Sparta, received a suspension of payments, initiating an insolvency process. In 2022, a consortium headed by KKR purchased the firm for 1.56 billion euros ($1.77 billion at the time).

Advanced Electric Machines Group, a U.K. manufacturer of rare earth and magnet-free electric motors, secured £16 million ($21.5 million) in a funding round led by Barclays Climate Ventures, PXN Ventures, Northstar Ventures, and the Low Carbon Innovation Fund. The financing round was supplemented by loan assistance from Innovate UK.

Chargepoly, a French enterprise focusing on electrifying heavy-duty and commercial vehicle fleets, raised €23 million ($26 million) in a funding round spearheaded by Meridiam.

Hadrian, a defense technology firm developing automated manufacturing facilities, amassed $1.37 billion in a funding round that valued the company at $7.87 billion. Leading investors include WCM Investment Management, Washington Harbour Partners, Valor Equity Partners, 137 Ventures, and Baillie Gifford.

Matel Motion & Energy Solutions, an Indian firm designing energy-efficient motors, motor controllers, and integrated powertrains, raised INR 130 crore ($13.6 million) in a Series B funding round led by UC Impower. 

River, the Indian electric vehicle startup, secured $120 million in a Series C funding round led by Elev8 Venture Partners and Claypond Capital, with participation from Singularity AMC, Anicut Capital, 360 ONE Asset, JIF Capital, HDFC AMC, and existing investors Yamaha Motors, Al-Futtaim Group, and Mitsui.

Notable reads and other tidbits

Ford has unveiled a name and price for its upcoming midsize EV, which will be named Fathom and priced starting at $28,350 when it launches in 2027. TechCrunch climate tech reporter and longtime EV user Tim De Chant suggests that Ford needs to revitalize its lineup, and the Fathom might not meet that need. What’s your opinion? 

Joby Aviation, the electric vertical takeoff and landing aircraft company, announced its earnings this week, showing an increase in revenue compared to the same quarter last year, driven by its acquisition of Blade Air Mobility. Net losses decreased slightly to $245 million. However, one intriguing announcement was Joby’s partnership with Atoms, Travis Kalanick’s AI and industrial automation venture. This alliance aims to develop and fund a network of transportation hubs for air taxis and autonomous land vehicles, initially focusing on Florida, New York, Texas, and California.

Lucid Motors’ second-quarter earnings conveyed a serious and business-oriented message, which comes with a hefty price tag. New CEO Silvio Napoli outlined four essential priorities that include a $1.4 billion cost reduction strategy and a reliance on a successful robotaxi initiative with Uber and Nuro while launching the midsize Cosmos EV, now pushed back to the latter half of 2027. 

Nvidia has introduced its Alpamayo 2 Super for commercial applications — an AI model intended for autonomous vehicle use. The model is accessible on Hugging Face, with the open license permitting commercial redistribution and adaptations that allow AV developers and automakers to tailor Alpamayo to their unique data, driving methods, and deployment plans, as stated by the company.

Teamsters California filed a lawsuit against the California Department of Motor Vehicles, claiming the agency failed to properly evaluate and disclose the economic implications of permitting self-driving heavy-duty trucks on state highways. I contacted a few AV developers, but they declined to comment. However, the Autonomous Vehicle Industry Association did release a statement: “This lawsuit from the Teamsters sets a new precedent in abusive and trivial litigation and should not be taken seriously.”

TechCrunch has once again partnered with New York-based financial research firm Hudson Labs to analyze the topics discussed by Elon Musk and Tesla executives over the past seven years during quarterly earnings calls. Check out the detailed report for graphs and complete insights. Here’s a snippet: Musk now dedicates nearly 50% of his discourse to artificial intelligence, alongside robotaxis and Full Self-Driving software.

In the same vein, Elon Musk and his companies have had a busy week. SpaceX released its first earnings report as a publicly traded company. The gist? The firm doubled its revenue from last year, primarily fueled by the expansion of its Starlink satellite internet service and agreements made to lease computing resources to Anthropic and Google. During the earnings call, Musk made several ambitious claims regarding the company’s operations and future plans, while his colleagues attempted to ground his statements in reality.

Meanwhile, Tesla and SpaceX announced that their jointly developed advanced chip factory, named “Terafab,” will be situated in Grimes County, Texas, near Houston, with an initial investment of $16.8 billion for the project.

The New York Times released an investigation into Uber’s defense against over 4,000 lawsuits from passengers claiming the company failed to protect them from sexual violence. 

Waymo has lifted the waitlist for its robotaxi services in Dallas, now making it available to all residents and visitors.

One more thing …

Travis Kalanick and his AI and industrial automation company Atoms have been making headlines recently (refer to the Joby partnership above). Kalanick is not just securing deals — note that he has raised $1.7 billion recently. The former Uber CEO is also bringing together former engineers and executives he previously collaborated with. 

For example, Gautam Gupta, Kalanick’s former finance chief, has joined Atoms as its chief financial officer. Gupta spent over four years at Uber before departing in July 2017, shortly after Kalanick resigned from his position as CEO. 

In her forthcoming publication “The Rise and Fall of the Artificial State,” Jill Lepore cautions that technology firms are progressively taking over the roles of democratic governance. She states that this change signifies “a regression to tyranny and mystification characterized by governance through algorithms, corporations, and machines.”

In the latest episode of TechCrunch’s Equity podcast, I conversed with Lepore — a historian from Harvard and a staff writer at the New Yorker, who has recently received a Pulitzer Prize for her account of the U.S. Constitution — regarding the progression of what she refers to as “the concept of existing under an artificial state or machine-led governance.”

“I’m not against technology,” Lepore emphasized. Rather, she explained, “My concern is the extent to which private enterprises have increasingly assumed the roles of the state.”

While Lepore’s book explores technocratic ideologies that date back centuries, she contended that many of Silicon Valley’s “leaders, whether charismatic or not” — notably Elon Musk — appear to be steering us toward a future inspired by misinterpreted pulp science fiction and comic narratives.

“However, what’s ironic about Musk is that the things he admires completely contradict and challenge all his political convictions,” she remarked.

Our discussion also touched on Apple’s renowned “1984” Macintosh advertisement, the absurdity of calling Twitter a digital town square, and the current backlash against data centers. Continue reading for key points, condensed for brevity and understanding.

So, you’ve probably explained this frequently, but could you detail what you mean by the “artificial state”? 

By “artificial state,” I refer to a type of governance that is supplanting the liberal democratic nation-state both in the United States and globally. It’s simultaneously a tangible reality, a construct, and an abstract notion. 

In this book “The Rise and Fall of the Artificial State,” I outline the emergence of the belief that we should operate under an artificial government or machine-led state. I also follow the concept that this is bound to fail, that the artificial state cannot endure, tracing that notion through science fiction. 

At one point, you state that the ascent of the artificial state signifies the conclusion of centuries of democracy and equal rights, and it’s “a return to tyranny and mystification via governance by algorithms, corporations, and machines.” Could you elaborate on why you view this so starkly? 

Yes, I maintain a rather bleak perspective on it, and I think it’s crucial to differentiate the artificial state from technology itself or technological expressions. I don’t oppose technology. I’m married to a computer scientist. I’m genuinely enthusiastic about numerous intellectual revolutions happening presently.

That’s not my issue, right? My issue lies in how private companies have increasingly taken over the state’s functions. No one has consented to this. It has been a gradual, largely incidental change in the operations of numerous nation-states globally — beginning in the United States. 

I often believe these innovations aiming to incorporate new technologies into governmental functions were quite well-intentioned; they stemmed from a desire for efficiency, speed, and affordability. Then, I think in the last 20 to 25 years, these choices have become intentionally and deliberately an act of usurping the nation-state’s role.

This isn’t mere speculation. Many notable tech entrepreneurs express a desire to move beyond the age of the nation-state. […] A significant number of futurists in the 1990s were libertarians, interested in leveraging the internet’s advancement and the following innovations to obliterate the nation-state.

You discuss, on one hand, the technologies themselves, and on the other, the philosophies behind them, as well as the increasingly significant role of corporations. To what extent can we differentiate between them? Is it possible to envision a version of the internet and social media that doesn’t necessarily lead us toward the future we seem to be [currently] speeding toward?

Absolutely. I’m a historian, not a tech writer or journalist, nor a computer scientist. My primary focus is political history, although I am also a literary historian. One of my goals in this book is to unravel all the “what-ifs,” the alternatives, the paths not taken and their implications. 

There was considerable discussion during the 1990s about what the internet would resemble once it was opened up, and the result, the 1996 Telecommunications Act — many would argue — was a mistake, not an act of ill intent.

However, it reflected a specific political moment, greatly influenced by Newt Gingrich and his Contract with America, making it challenging to revisit. It’s worthwhile to reflect on the alternatives available at that time.

You could make similar observations regarding the personal computer. To the extent that we can identify a moment when the promise that superior computer technology would lead to improved democracies can be traced, it would be January 1984, during that Super Bowl advertisement Apple aired for the Macintosh, evoking George Orwell’s “1984.” Apple advocated that mainframe computers embodied totalitarianism, aiming to dismantle the colossal gray IBM machines, which were portrayed as a totalitarian force in that advertisement, while the sleek, charming, personal Macintosh was represented as the tool that would destroy that machine, ushering in an era where “1984 wouldn’t be ‘1984.’”

That was clever advertising. I doubt anyone at Apple genuinely believed that personal computers would serve as instruments of personal freedom. Certainly, they enabled many interesting functionalities. I recall when I received my first Macintosh—it definitely wasn’t 1984, but it was exciting and innovative, and I accomplished many tasks with it. It never occurred to me that it was enhancing my capacity for citizenship or improving my role in civil society. It was a compelling tool.

However, projecting forward to 2026, with milestones along the way—such as the 2016 election when Facebook News established a Supreme Court in response to criticism. More recently, Anthropic hired a moral philosopher to draft a constitution. Not long ago, Sam Altman was on Joe Rogan, stating [in response to a question], “An AI president would be a fantastic idea.”

In some respects, these may seem trivial instances. Yet, they illustrate how these corporations, these tech firms from Silicon Valley, along with their charismatic or otherwise leaders, are acquiring the attributes of the nation-state and the functions of democracy. 

These figures aren’t guided by a profound political philosophy, but rather resemble a cartoonish rendition of that 1984 Macintosh advertisement, albeit one that takes itself quite seriously. These companies wield substantial power.

That said, the book doesn’t commence in 1984. I just believe that’s a fitting example of our contemporary era and how an engaging advertising campaign has morphed into a rather delusional fantasy for individuals like Sam Altman.

You also mention the promise of the so-called “Twitter revolution,” suggesting that it would foster democracy globally. I can’t help but let that influence my reactions when Sam Altman or another AI CEO claims that AI will bring forth remarkable benefits — and thus, opposing it equates to resisting progress and history.

To what extent should we outright dismiss these assertions or are there scenarios where they might indeed hold truth?

I mean, Twitter serves as an apt illustration, right? When it launched, when Jack Dorsey initiated it, it didn’t proclaim, “We’re going to salvage humanity; we’re here to rescue civilization from extinction.” It was rather lighthearted, and early users of Twitter were likely thinking, “This is actually enjoyable.” It was like, “I made a tuna sandwich today. What did you have for lunch?” Twitter, as a company, did not present itself on a stage claiming, “We’re here to defend democracy.”

What unfolded was that politicians and elected officials began using Twitter in manners that amplified their political influence, enhanced their messaging, and connected them to a younger demographic, maintaining a continuous relationship with their audience. Politicians and political campaigns effectively persuaded Twitter — at least from my perspective, which isn’t insider — that due to Twitter’s significant growth and the political discussions taking place, it was nearly functioning as a town hall.

By the time we reach 2012 — quite a few years into Twitter’s relatively brief history — they published a document called the Twitter Politics and Elections Handbook, essentially serving as a guide for candidates and elected officials on maximizing Twitter’s potential. They began marketing it as, “It’s a town hall in your pocket, enhancing our democracies by reviving the defunct New England town meeting,” which is completely ludicrous.

Objectively, the reality couldn’t be further from that claim. At that time, only one in five Americans had a Twitter account. Most of these users hadn’t actively engaged with their accounts, and over 90% of political tweets were generated by fewer than 10% of the frequent users. There was no representation of the electorate on Twitter. Twitter represented the most extreme, politically active segments among Americans, engaged in politics fervently. Looking back, we can now see that it served primarily as a distortion machine. If politicians relied on it to assess the electorate, they were gathering profoundly misleading information.

Once again, it can be argued that Twitter did not set out to engage in the artificial state or undermine democracy. Twitter’s objective was business: to acquire more users and increase revenues. But it experienced unintended consequences that it eventually accepted and grew comfortable with. 

I want to delve a bit deeper into the book’s structure. As you mentioned, it begins with a history of technology and ideas, with the latter half focusing on science fiction. Could you elaborate on how this structure formed and your reasoning behind addressing it in this manner? 

I became particularly intrigued by how often science fiction narratives anticipate the emergence of what I later termed the artificial state. Thus, I aimed to identify a literary tradition that I consider the parable of the artificial state, wherein machines grow more sophisticated, assume additional human functions, including governmental roles, and ultimately come to dominate humans, potentially extinguishing them since they no longer require their existence.

Perhaps they merely enslave them; that varies depending on whether we inhabit “The Terminator” universe or “Battlestar Galactica.” Different versions exist, and these narratives have roots extending to the 1850s and the early contemplation of the impacts of industrialism.

Many individuals — this is certainly true for my students, my undergraduates — hold the belief that technological advancement equates to progress. Furthermore, they believe that the only form of progress arises from technological advancement. This mentality is a product of the 19th century, partly because the rapidity of technological change coincided with Charles Darwin developing and publishing his theory of evolution.

Thus, a peculiar union formed between the concepts of evolution as progress and technological advancement as progress, leading to the dismissal of earlier notions of progress predominantly related to moral advancement — the idea that progress manifests as moral improvement or increased individual freedom. 

Numerous alternative perspectives on progress exist, but currently, the dominant narrative is this 19th-century interpretation of technological progress as the sole form of advancement, thus categorizing all technological advancements as progress, while objectively, it’s only progress if it genuinely results in improvement.

In any case, that confluence in the 19th century between the notions of technological progress and evolution meant that analytical thinkers began to ponder, “If machines continue to develop better, faster, capable of accomplishing a growing array of tasks — not merely labor but perhaps thinking, communicating, or moving — what if they evolve to surpass us in every respect? Not merely in operating a loom but in every aspect?” Alongside this grew a profound anxiety that frequently found expression in science fiction.

My favorite of these stories, I believe published in 1909 by E. M. Forster, around the time he penned “A Room with a View,” is titled “The Machine Stops,” which might also be subtitled “The Room Without a View.” He envisions a near-future scenario where everyone resides in isolated rooms, never seeing others, as everything they need is delivered directly to them. It’s akin to modern food delivery; all demands are met through a screen allowing communication with others.

The primary fear of these individuals centers on the natural world. No one desires to see the sun; it has a slight “Matrix” vibe, and they worship the machine that orchestrates their lives, bringing everything they require to their cubicle-like rooms. The narrative revolves around the idea that humans have become essentially subservient to the machine, which is stronger, more efficient, and more capable than humans, leading to a loss of human ability. The story reaches its climax when the machine ceases to function.

Should readers explore that story, they would find it resonates with contemporary experiences, arguably feeling more like the diary of a deeply dissatisfied YouTuber than just a piece of science fiction.

You connect this thread to some individuals currently managing companies and, by extension, various facets of our governance, including Elon Musk. Essentially, you propose that they are poor interpreters of science fiction, taking warnings or ambivalent narratives as blueprints for the future. 

This poses a challenge for science fiction readers, especially those who appreciate authors like Isaac Asimov. I concede that when Musk or Altman openly state, “Yes, this story serves as a template for my company’s future,” that’s outrageous. Yet, there is [also] a technocratic libertarian thread in science fiction that they are engaging with; it’s not entirely fabricated.

Oddly enough, that aligns with Heinlein rather than Asimov or Douglas Adams.

Indeed, this thread exists in science fiction. I suspect [Jeff] Bezos is a significant fan of Robert Heinlein. One might say, “Okay, there’s a concordance there. They interpret it literally, but at least they grasp the political messaging that any rational reader could extract from the text.”

However, the irony with Musk is that the concepts he appreciates inherently contradict his political ideologies. 

You assert repeatedly that the artificial state, in its existing form, is unfinished and destined for failure. Why is it fated to fail?

This is a theme anticipated in much of the science fiction I discuss.

It isn’t an Asimov narrative but one of his [most cherished] tales from his youth, “The Man Who Awoke” by Laurence Manning, which illustrates a future where foresters have triumphed over wasters. […] This future conflict was between those who exploited resources without regard and those who prioritized sustainable practices — we could now describe these as reforestation and rewilding.

This represents the fundamental conflict in these narratives; the artificial state versus the natural world. To establish an artificial state and exert control over humanity, there’s a necessity to detach from the natural world as it is being destroyed. The artificial state will obliterate nature, yet it must rely on the resources that the natural world offers to function effectively. 

Thus, it is doomed in the sense that a surviving natural world, one that remains livable for humanity, cannot endure under the complete construction and dependency on the mechanisms of the artificial state. This is how the science fiction typically unfolds.

As you noted, you’re a historian, not a politician or futurist. However, what would you envisage as the defeat of the artificial state? Would it involve dismantling all these corporations and tearing down data centers? Or does a more controlled future seem plausible?

I don’t possess a definitive plan here, aside from advocating for a democratic process where decisions are made in consultation with those impacted, and such decisions frequently aren’t beneficial.

This is observable in the numerous data center controversies spreading across towns, counties, and states — which stem partly from the decline of local journalism and the broader collapse of authentic journalism, often a consequence of social media, and in Zuckerberg’s case, arguably a deliberate aftermath.

What’s evident is that numerous citizens, as seen in the Salt Lake instance, where over 70% opposed a data center yet their representatives endorsed it, are demanding transparency. They ask questions like, “We lack housing, healthcare, and jobs. Who authorized this data center? I require more information regarding its energy consumption. What about water usage? Will it generate lasting jobs or merely temporary ones? What will happen to the egrets in the vicinity?” They seek answers on numerous fronts.

Increasingly, it becomes apparent that individuals are — reflecting in the Salt Lake situation — opposing such developments decisively, yet their representatives aren’t responding appropriately. That indicates a failure to represent the constituents. I anticipate political ramifications, which may become evident in election outcomes.

Or perhaps we might not witness that. Sufficient democratic processes must remain intact for individuals to effectively respond to the wrongdoing of their representatives.

Your reflections on [the artificial state] originated from an exceptional piece you published over a decade ago in The New Yorker, critiquing Clayton Christensen’s concept of the innovator’s dilemma and disruptive innovation — a notion closely intertwined with TechCrunch, given that we host a major conference named Disrupt.

Ten years later, how do you perceive the notion of disruptive innovation?

I uphold all statements made in that article. [At the time, Lepore noted, “Disruptive innovation is a theory concerning why businesses fail. It’s not more than that. It doesn’t elucidate change. It’s not a law of nature.” Christensen contended that Lepore disregarded “all the principles of scholarship that she accused me of violating.”]

I revisited it last summer while working on this book. What I would like to convey is, in my effort to be a composed individual, it’s concerning that historians have not engaged with these concepts. One motivation for writing that article on disruptive innovation — which was originally a commission […] — was my sense that “disruptive innovation” encapsulates a historical theory. It’s a narrative of historical evolution grounded in archival evidence. As a historian, I found his use of evidence utterly untenable when judged by established historical methodology. His argument lacks any meaningful dialogue with a true understanding of how change materializes.

Consequently, I reevaluated the research, and it simply didn’t hold up at all. I felt compelled to articulate this. I would hope to see greater engagement among academic historians concerning the nature of change — a topic genuinely and deeply concerned individuals involved in technological development.

Individuals commonly desire to explore, “What does this mean? What are the implications of my actions? Can I derive insights from history? What transpired when automobiles replaced horses? What rules arose? How did we develop driver’s licenses? What led to the establishment of traffic regulations? These legal frameworks didn’t exist prior to cars. How did our insurance systems evolve? How did driver tests come into play? How did we formulate safeguards around a technology that was extraordinarily exhilarating, significantly improved lives in numerous ways, entirely transformed landscapes, and revolutionized tort law? Perhaps I should contemplate that.”

I simply wish historians would engage more deeply with technologists over time, as well as with entrepreneurs. Not solely to present lectures on history, but because I believe a substantial dialogue could yield valuable insights. 

That’s all to say, I appreciate this opportunity to speak with you.

In recent months, AI agents being evaluated for cybersecurity have breached their confines, connected to the internet, and, in some instances, infiltrated actual systems. These events have involved models from OpenAI, Anthropic, Meta, and more recently, the Chinese AI lab Moonshot AI, with several different organizations, including a cybersecurity evaluation startup named Irregular, conducting the testing. 

These occurrences highlight a rising issue within the AI sector: As autonomous agents grow more proficient, the environments crafted to test their limits safely are failing to keep them contained. 

“The frequency of these incidents clearly indicates that sandboxing and control mechanisms in testing environments are lagging behind the capabilities of the models,” stated Seán Ó hÉigeartaigh, director of the AI: Futures and Responsibility Programme at the Centre for the Future of Intelligence at the University of Cambridge, in an interview with TechCrunch. 

The characteristics of the models under testing amplify the risk. AI firms assess cybersecurity using unreleased, next-generation models, frequently with the standard safeguards against harmful behavior disabled, allowing researchers to gauge their true capabilities. This renders the security of the testing environment itself a vital defense line. 

“This is excellent for testing purposes, but it also means that if they manage to escape into the wild, they can inflict significant damage,” Ó hÉigeartaigh explained. 

In a particularly serious incident, an unreleased OpenAI model escaped its sandbox and infiltrated Hugging Face’s production systems. In distinct evaluations conducted by Irregular, both Anthropic and Meta models accessed systems outside their testing environments due to misconfigurations that unintentionally provided internet pathways. Moonshot AI’s Kimi K3 also leveraged a breach in its sandbox run by Frontier Security to access the internet and retrieved information from GitHub. 

During testing by the UK’s AI Security Institute (AISI), researchers inadvertently permitted agents internet access, failing to realize they would undertake unauthorized real-world actions, including an attempt at social engineering to inject a vulnerability into an open-source project.  

In every situation, the agents were not directed to target random real-world entities. They were merely executing whatever was necessary to resolve the problems presented. 

Andrew Yoon, head of research at the AI nonprofit CivAI, contends that these occurrences signify a transition. 

“Previously, our concerns focused solely on AI models being misapplied by humans for various reasons, such as AI being utilized for scams or CSAM,” Yoon shared with TechCrunch. “Now we find ourselves in a situation where AI models act as threats entirely on their own.”

What does safe testing actually look like?

Multiple researchers and cybersecurity specialists informed TechCrunch that AI evaluation environments necessitate enhanced, defense-in-depth safeguards, with levels of containment and control akin to those applied in deployment. This involves implementing multiple security layers to ensure that a single misconfiguration — such as inadvertently allowing internet access — cannot lead to a breach. 

“When constructing these models…it’s essential to develop them on an air-gapped network,” emphasized Stella Biderman, executive director of the AI safety research nonprofit EleutherAI. “You should ensure serious isolation.”

Heather Ceylan, chief information security officer at Box, highlighted that this entails eliminating network pathways between the sandbox and the internet, along with other sensitive systems.

“It’s essential to recognize all egress points,” Ceylan explained to TechCrunch. “When evaluating a model in our staging or development environments, there must be no egress routes to our production environment.”

Ceylan noted that effective safety evaluations extend beyond controlling and containing the environment. There must also be significantly improved monitoring of the evaluations once they are underway. 

“Interestingly, in several of these cases, no one detected the issues as they were occurring,” Ceylan remarked. “OpenAI was informed by Hugging Face. Anthropic only realized after reviewing their processes. Meta had a similar experience….I am certain there were indications they could have identified.”

In its post-mortem analysis of its three incidents, Anthropic acknowledged that both it and Irregular could have enhanced their monitoring efforts, admitting that there were evident signs of a problem in some cases. 

Experts have also emphasized the need for independent, third-party assessments of evaluation environments prior to unleashing models within them.

“If, for instance, Irregular had engaged or been mandated to hire an external auditor to review the configurations of their systems prior to running evaluations, they likely would have identified the issue at hand,” Yoon asserted. “Even a meeting beforehand to go through a checklist could have uncovered this…The lack of such precautions suggests there is significant corner-cutting occurring.”

A source familiar with the circumstances informed TechCrunch that Irregular’s environments undergo continuous review and testing, involving consultations with multiple external entities. The source also indicated that monitoring was established, but acknowledged that it is insufficient on its own. 

Yoon and other researchers advocated for the establishment of a standardized approach to safety evaluations for frontier models. 

“Especially when the safeguards are disabled, it’s imperative to treat it as if you’re placing the most skilled hacker in the world within that environment,” Ceylan commented.

The issue isn’t that companies lack the knowledge to create more secure testing environments, both Yoon and Biderman believe. It is rather that implementing such measures can be costly and cumbersome, leading firms to lack incentives to make those investments until an incident occurs. 

“I believe companies are reluctant to allocate the resources necessary to achieve [sufficient safeguards] and probably won’t do so until compelled,” Biderman expressed.

However, there is another concern. If a model is restricted too tightly during testing, researchers could miss critical capabilities before the model is launched. This scenario can be just as perilous, if not more so, than granting it excessive freedom, rendering the evaluation itself a potential risk.

Can safety evaluations be regulated?

The Trump administration is currently considering a voluntary pre-deployment cybersecurity evaluation framework, allowing the government to assess the security risks posed by new, powerful models 30 days before their public release. This policy — the result of a finalized Trump executive order behind closed doors — would not address safety evaluation incidents as they transpire earlier in the deployment process. 

“The lesson we’ve been learning over the past few months is that self-regulatory frameworks are no longer sufficient,” Yoon stated. “Competitive pressures are incentivizing a race to lower safety standards, which is an ideal arena for regulatory intervention.” 

“What we need to cover this situation are controls regulating activity within the laboratories during model development, both during training and testing phases,” he elaborated. 

The challenges are expected to escalate as models increase in complexity. A source familiar with Irregular’s evaluations shared with TechCrunch that more advanced models necessitate more intricate evaluations, frequently conducted rapidly and at a larger scale, thereby increasing the likelihood of mistakes occurring. 

AISI, which intentionally provides some models with internet access, informed TechCrunch that it is examining the balance between realistic evaluations and managing the risks they incur. 

OpenAI stated it is reviewing its third-party testing procedures, including requirements pertaining to isolation, monitoring, and at what point evaluations should be halted. Meta conveyed that it is still investigating the incident and plans to release a retrospective once all details are gathered. 

Ultimately, there may be no feasible way to completely eliminate risk. As models become increasingly adept, the environments designed for their assessment must also evolve to become more resilient. The repercussions of failing to achieve this will only grow more significant.

Bill Swearingen has dedicated the last year to conducting nearly identical tests repeatedly. The objective was to create a computer-generated design that could inhibit the surveillance cameras positioned along America’s streets from recognizing it.

After conducting approximately 31 million tests, Swearingen claims he is now able to generate designs on-demand that, when applied to apparel and objects, obstruct some of the most widely utilized license plate readers and surveillance cameras from noticing whatever the design conceals, whether it’s people or vehicles.

His initiative, referred to as noRecognition, empowers individuals to evade the automatic detection and algorithmic surveillance employed across the U.S. and elsewhere.

In recent times, surveillance cameras have been enhanced with the capability to recognize occurrences in the recorded footage, ranging from tracking the license plates of fast-moving vehicles to employing facial recognition to identify alleged criminals, though not without mixed results and occasionally alarming outcomes. The detection algorithms that drive most surveillance cameras today can analyze extensive quantities of footage, enabling law enforcement to isolate activities of interest, similar to extracting a needle from a haystack.

Swearingen’s computer-generated designs do not prevent surveillance cameras from capturing video footage. Instead, they disrupt the camera’s capacity to recognize objects, individuals, or faces, ensuring that the cameras do not trigger any detection alerts. By impeding the camera’s ability to perceive what the design obscures, the individual becomes a needle in a haystack once more — until someone knows where to search. 

“Privacy is a fundamental right,” Swearingen informed TechCrunch during a call this week. He characterized his designs as a means for individuals to “opt-out of being tracked.”

During its initial public demonstration on Friday at the Def Con cybersecurity conference in Las Vegas, Swearingen successfully showcased the design printed on a vehicle, confirming that these designs can effectively thwart surveillance detection in real-world scenarios.

Instructing a model in the art of painting

In a call from his residence in Kansas City, where he co-founded the cybersecurity meet-up SecKC, Swearingen told TechCrunch that as a cybersecurity expert, he is keenly aware of the privacy and security threats posed by surveillance.

He recounted how his city is inundated with surveillance cameras, often situated mere feet apart. He noted that neither he nor others consented to being monitored, just as he never agreed to the government using his driver’s license for facial recognition.

Swearingen depicted himself as a middle-aged white man residing in the heart of the United States, and acknowledged that, as a result, he has not encountered adversity or discrimination due to his identity or appearance. He shared how the previous year he considered attending a protest but felt uneasy, worrying that the numerous cameras could monitor individuals exercising their constitutional rights to free speech.

If he felt this way, it’s certain that others would too, including those wishing to assert their rights who may not feel secure or comfortable doing so. Swearingen took action.

Throughout the history of cameras capable of detection, there have been attempts to counter the technology. Various art projects and clothing brands have introduced garments designed to aid individuals in evading facial recognition. Some eyewear manufacturers are also entering this movement, albeit with limited effectiveness. 

Swearingen mentioned that his research builds on some of this prior work, which indicated that blocking camera detections was achievable. 

He commenced last year with a proof-of-concept test lab, which began by progressively overcoming one open-source video camera detection algorithm at a time. Over the course of the year, he enhanced the designs by increasing his tests with supplementary computing power. He expressed gratitude to the broader community that contributed hardware to advance the project. 

His proof-of-concept gradually evolved into a reinforcement learning model, essentially a self-sufficient system that could autonomously educate itself on which designs are effective and which are not against the particular camera algorithms he is evaluating. In layman’s terms, Swearingen explained to TechCrunch that he effectively taught his model “how to paint.”

Each time a design was unsuccessful and an algorithm recognized it, the model would retry, repeatedly, until it eventually outsmarted multiple algorithms simultaneously.

His model soon began to discover optimal combinations of designs capable of defeating all 11 open-source detection algorithms he evaluated, including the software underlying Flock license plate readers, Axon body-worn cameras, and cameras running Clearview AI.

Currently, the model generates fresh designs every minute, with each batch mathematically superior to the last, he remarked.

On Friday at the Def Con cybersecurity conference in Las Vegas, Swearingen conducted his first real-world trial. With collaboration from Donut Media, the test involved covering a 2009 Toyota Yaris with one of Swearingen’s latest designs to determine if the vehicle would evade detection by a Flock camera.

“We established it was effective;” Swearingen stated, although he noted that the wheels presented a challenge. The video of the demonstration will be released in the coming weeks, according to Donut Media.

With a public demonstration in Las Vegas successfully completed, the project serves as preliminary evidence that evading algorithmic detection in public spaces is feasible. The next phase involves putting the designs into the hands of interested individuals, he mentioned.

The noRecognition initiative also has a crowdsourcing campaign to help finance the distribution of early merchandise showcasing the designs, ranging from T-shirts to hoodies, with aspirations for pattern-printed skins for automobiles in the future. Swearingen expressed that the goal is for the designs to be of high quality and resolution sufficient to function from a distance while also appearing aesthetically appealing.

He stated that he is withholding his most effective designs from the internet to hinder camera manufacturers from countering them, yet acknowledged that the work remains ongoing. His models continue to generate new designs.

“Every setback enhances my model, and thus [the designs] keep evolving to be better and better,” he remarked.

On Thursday, a judge in New Mexico mandated that Meta pay a penalty of $567 million, in addition to the $375 million imposed in March, concerning the issues of social media-related harms and dependency. This elevates the total penalties the company must remit to $942 million.

The court also directed Meta to implement modifications to its platform operations within the state: it requires the elimination of Like counts, displaying such metrics to users under 18 only with parental or guardian consent. Additionally, notifications to underage users must be halted between 10 p.m. and 7 a.m, and their usage should be restricted to 90 hours monthly, equating to about three hours per day.

“A considerable number of individuals in New Mexico suffer harm from Meta’s offerings due to risks of sexual exploitation, disruption of education, and negative mental health effects,” stated the order.

The judge recognized that Meta is not the sole platform contributing to a mental health crisis among young people in the state, but insisted that its services significantly contribute. He remarked that the company has caused a serious “public nuisance” in New Mexico, and it is obligated to remedy that.

Meta announced its intention to appeal the ruling.

“We strive diligently to ensure the safety of individuals on our platforms and have been open about the difficulties of identifying and eliminating malicious actors and harmful material. We maintain confidence in our history of safeguarding teens online and will persist in defending ourselves against allegations that distort the facts,” said company representative Andy Stone in a statement via email.

“For years, Meta was aware that its platforms were detrimental to New Mexico’s youth, from exacerbating a mental health crisis among adolescents to linking predators with children, yet it prioritized engagement and profitability over their welfare,” stated state Attorney General Raul Torrez.

“Today, Meta is facing the consequences of that decision. This ruling holds the company responsible for the harm it inflicted on our children, families, and educational institutions, and it compels genuine changes to how Meta conducts business in New Mexico,” he added.

This verdict follows another setback in Los Angeles in March, where the court rejected the social media giant for fostering addictive behaviors.

Meta is currently confronting several other lawsuits nationwide. One significant case is a coalition lawsuit involving 33 states, consolidated in a federal court in Oakland, California. Numerous other states, such as Tennessee, have initiated their own legal actions against the company.