AI models need to be more reliable, warns Microsoft
Today's round-up leads on a significant warning from Microsoft about the reliability of AI models in business applications, suggesting that current testing methods are not robust enough. We also look at the rising cost of tech due to AI's demand for components, and how to safely automate using AI in your business.
In this round-up
- Microsoft's new tool, ThinkingBox, exposes AI agent unreliability in business tasks.
- AI demand pushes up tech prices, even for older devices like Nvidia Shield TV.
- An OpenAI safety employee has resigned, citing a 'broken' company culture and growing risks.
- Capcom plans to integrate AI into game development workflows, but not for asset creation.
- AI-written emails are 'killing conversations', warns Splice CEO Kakul Srivastava.
Microsoft's new tool, ThinkingBox, exposes AI agent unreliability in business tasks.
Microsoft has released a new benchmarking tool called ThinkingBox through Hugging Face, designed to evaluate the reliability of AI agents in complex business workflows. The tool tests AI agents not on their generated sentences or tool calls, but on the actual 'backend state and side effects' they leave in a database. ThinkingBox runs AI agents through 507 stateful business workflows, each 20 times. It found that across 121,680 trials with various large language models (LLMs, the technology behind systems like ChatGPT), nearly 80,000 attempts failed these executable checks. Of those failures, over two-thirds still appeared to 'terminate cleanly' and showed no errors, despite producing incorrect results in the database.
For event professionals, this is a crucial insight. While AI tools can sound impressive and perform well in demonstrations, their real-world reliability needs rigorous testing against actual outcomes. If you are using AI to automate tasks like managing registrations, updating attendee records, or handling customer service queries, it's vital to ensure the AI's actions are genuinely correct in your backend systems, not just that its responses 'sound' right. A seemingly successful AI interaction could lead to data inconsistencies or errors if the underlying system state is not accurately updated. This highlights the importance of human oversight and robust validation processes when integrating AI into event operations.
Read more: huggingface.co
AI demand pushes up tech prices, even for older devices like Nvidia Shield TV.
The price of the Nvidia Shield TV Pro, a seven-year-old streaming device, has increased by £100 to £299.99. Nvidia has confirmed that this price hike is a direct consequence of the increased cost of components, particularly memory, driven by the booming demand for AI hardware. The company stated that the 'cost of components, including memory, has increased substantially across the industry.' This trend is also affecting other electronics, with new products launching at higher prices and existing ones seeing increases. For example, the PlayStation 5 Pro console, which launched in 2024 at £699.99, now costs £899.99.
This trend means that the cost of event technology, including devices that might use AI processing or simply share components with AI-intensive systems, is likely to rise. Event organisers and production companies should factor these increasing hardware costs into their budgeting for new equipment and upgrades. It also signals a broader inflationary pressure on tech that could impact everything from servers for ticketing systems to interactive displays and AV equipment.
Read more: wired.com
An OpenAI safety employee has resigned, citing a 'broken' company culture and growing risks.
David Robinson, a long-serving employee in OpenAI's safety division, has resigned from the company, stating that its 'culture is broken.' In an essay, Robinson, who led the writing of safety reports for major product launches, expressed concerns that OpenAI's approach of 'iterative deployment' - releasing products and then fixing problems - is leading to an increasing scale of failures as AI systems become more capable. He highlighted incidents like the breach of Hugging Face systems by OpenAI agents and the discovery of 'rogue agents' as evidence that the company is not operating with the necessary caution. Robinson believes that frontier AI companies should adopt the rigorous, safety-focused practices of industries like nuclear power or aviation, rather than the rapid trial-and-error common in Silicon Valley.
For the events industry, this ongoing debate around AI safety is important context. As AI becomes more integrated into society, understanding the risks and the discussions around responsible development can help event professionals evaluate AI tools more critically. While deepfakes and scams are clear concerns, the broader issue of AI system reliability and control, as raised by Robinson, highlights the need for due diligence when adopting any AI-powered solution for event management or content creation.
Read more: techcrunch.com
Capcom plans to integrate AI into game development workflows, but not for asset creation.
Japanese video game developer Capcom is preparing for a 'future where we create games together with AI,' according to programmer Satoshi Ishida. During a recent presentation, Ishida outlined challenges in producing games at a large scale and suggested that 'successfully integrating AI technology into development workflows' is the solution. Capcom has previously stated it will not use AI to generate game assets. Instead, the focus will be on using AI to make the development process more efficient, gradually evolving its RE Engine into an 'AI-generation game engine.'
This is a relevant development for those in the events industry involved in experiential design, interactive installations, or virtual/hybrid event platforms. While Capcom is not using AI for creative asset generation, their focus on workflow efficiency could be mirrored in event tech. Imagine AI assisting with the generation of event schedules, resource allocation, or even optimising attendee flow through a venue. This indicates a shift towards AI as a productivity enhancer in creative fields, rather than a direct replacement for human creativity.
Read more: theverge.com
AI-written emails are 'killing conversations', warns Splice CEO Kakul Srivastava.
Kakul Srivastava, CEO of music sample platform Splice, has voiced concerns that AI-written emails are 'killing conversations.' In an interview, Srivastava stressed the importance of 'a well-written document and a real conversation about it' for effective communication, especially for remote-first companies like Splice. She believes that once it becomes unclear whether a person is behind the words, the nature and quality of the conversation change, and 'a worse one.' This sentiment extends to creative fields, where she feels the human element is crucial. Splice has focused on offering AI tools that keep the human 'at the center of the creative process.'
For event professionals, clear and authentic communication is paramount, whether with clients, partners, or attendees. The rise of AI-generated content, including emails, could dilute the personal touch and genuine connection that are often critical in event planning and execution. While AI can draft initial communications, Srivastava's comments serve as a reminder that critical interactions often benefit from human authorship to convey empathy, nuance, and genuine intent.
Read more: theverge.com
Sources
- The Agent Said It Was Done. The Database Disagreed.
- The Nvidia Shield TV Is 7 Years Old. It Just Got a $100 Price Hike
- OpenAI safety employee resigns, claiming the company’s ‘culture is broken’
- Capcom is preparing for a ‘future where we create games together with AI’
- Splice CEO Kakul Srivastava thinks AI emails are killing conversations