Here's a neat thing I had ChatGPT Work with GPT-6 Astra (Max) do this morning: I live at <my address>. Figure out 5K and 10K running routes from me that loop from my house. Use OSM data. It worked for 27 minutes and produced exactly what I'd asked for, as both an embedded visualization and downloadable GPX file and GeoJSON files. Here's that 5K route: When I asked it how it had created the route, it replied: I used Nominatim to locate the address and Overpass to download local OpenStreetMap roads and trails , then calculated the loops locally. Frustratingly, the actual code it ran and exact details of what it did weren't visible to me in the ChatGPT UI. I see this lack of transparency is an anti-feature. By the time I thought to ask for a copy of the Python code it had used, ChatGPT was unable to provide it. This appears to be because the thread had been compacted. I think any LLM system that uses compaction needs to both preserve the pre-compacted text and make that text available via agent tool calls, to protect against this kind of problem. As for displaying the map to me, that used the visualize skill . It created a file called /workspace/el-granada-5k-share.html to embed directly into the ChatGPT UI. Here's a copy of that HTML , which starts like this: < div id =" eg-share-loop " > < div class =" viz-row " > < h3 > El Granada harbor loop </ h3 > < span class =" text-small " > 5.1 km </ span > </ div > < div id =" eg-share-stage " > </ div > < div class =" text-small text-muted " > Map data © < a href =" https://www.openstreetmap.org/copyright " target =" _blank " rel =" noopener " > OpenStreetMap contributors </ a > </ div > < style > # eg-share-loop { width : 100 % ; } # eg-share-loop # eg-share-stage { width : 100 % ; margin : 8 px 0 ; } # eg-share-loop . eg-share-map { display : block; width : 100 % ; touch-action : none; } # eg-share-loop . eg-share-map text { fill : var ( --foreground ); font-size : 12 px ; font-weight : 400 ; } # eg-share-loop . eg-share-label { paint-order : stroke; stroke : var ( --background ); stroke-width : 3 px ; stroke-linejoin : round; } </ style > < script type =" application/json " id =" eg-share-data " > { "route" : { "type" : "LineString" , "coordinates" : [ [ - 122.467425 , 37.4997753 ] . . . </ script > < script src =" https://cdn.jsdelivr.net/npm/[email protected]/dist/d3.min.js " > </ script > < script > (() => { const root=document.getElementById('eg-share-loop'); The <script type="application/json"> element contains the full geometry needed to render both the running route and the map itself, using D3, which is loaded from an allow-listed CDN location described in this section of the visualize skill : External resources The CSP allows only cdnjs.cloudflare.com , esm.sh , cdn.jsdelivr.net , unpkg.com , fonts.googleapis.com , fonts.gstatic.com , and fonts.bunny.net . Other origins are blocked and fail silently. Tags: geospatial , ai , d3 , openai , generative-ai , chatgpt , llms , skills , gpt-6-astra

Simon Willison LLMs11:56 PMHeat 72

ReadSource

Hacker News AI / 10:45 PM

Show HN: Generate a Claude Code or Codex agent fleet from one HTML file

HN 2 pts · 0 comments

ReadSource

Simon Willison LLMs / 4:39 PM

llm-gemini 0.34

Release: llm-gemini 0.34 New model gemini-3.8-flash for Gemini 3.8 Flash , with low, medium and high thinking levels. #146 Fixed async responses failing to record the resolved model version. Thanks, Charlie Tonneslan . #137 Google released Gemini 3.8 Flash (and 3.8 Flash Cyber, but that's available to "trusted defenders" only) today. Here are the pelicans for high, medium, and low. This is high: For comparison, here are the same pelicans generated using Gemini 3.7 Flash . Something I appreciate about Gemini Flash is that it's fast, cheap, and competent at things like HTML and JavaScript. I was messing around with it and prompted "make me a cool thing in html" and it built this , which is certainly a cool thing in HTML! Took 13 seconds, cost 1.8 cents. Your browser does not support HTML5 video. If you click through to the demo you'll see one more thing I built with Gemini 3.8 Flash. My markdown-svg-renderer tool lets me feed in the URL to a Gist with Markdown in and renders that markdown with fenced code blocks for SVG correctly rendered. I used Gemini 3.8 Flash (with my very basic llm-coding-agent coding agent plugin) to add support for HTML as well, so now any HTML blocks in the Markdown are rendered using a sandboxed iframe. Here's the transcript . Tags: ai , generative-ai , llms , llm , gemini , pelican-riding-a-bicycle , llm-release

ReadSource

Product Hunt AI / 3:31 AM

Sendra

Design emails in Figma, export HTML that works everywhere Discussion | Link

ReadSource

Hacker News AI / 11:31 PM

Show HN: Agent skill to create HTML docs for planning

HN 2 pts · 1 comments

ReadSource

The Verge AI / 12:00 PM

Slack is launching collaborative vibe-coding channels

Slack is introducing dedicated channels where teams can vibe-code together with AI agents instead of jumping between different tools and conversations. The Slack Code launch includes open, project-specific code channels with dedicated user tabs, alongside features that compare coding changes and preview HTML output before the project is shipped. "With Slack Code, when you have […]

ReadSource

Product Hunt AI / 7:55 PM

Doxy

Markdown & HTML editor, without the LaTeX pain Discussion | Link

ReadSource

Simon Willison LLMs / 2:16 PM

Claude's new system prompt really doesn't want to reproduce song lyrics

Anthropic publish the system prompts for their Claude consumer applications ( Claude.ai and the Claude mobile apps - sadly not for Claude Cowork or Claude Code). I love that they do this, and that they share not just the current prompts but historic changes to their prompts as well. They used to keep all of the prompts on a single page, but when I checked today I noticed they had re-arranged those prompts into an index page and then a page per model - here's the page for Haiku 4.5 for example, which has the original prompt from October 15th 2025 and an updated prompt from January 18th 2026. A neat thing about Anthropic's platform.claude.com/docs site is that it's designed to be usable by LLMs. You can add .md to any page to get back the content as Markdown - here's the system prompt index page and the Markdown prompts for Fable 5.1 . TL;DR: this makes it really easy to diff the prompts. Don't reproduce song lyrics Don't draw copyrighted characters or logos Tweaks to Claude's answering style The missing end_conversation guidelines Recommended substance support sites Reliable cutoff date of June 2026 How I'm tracking these prompts Don't reproduce song lyrics Let's start with the most interesting difference between Fable 5 and Fable 5.1 : There's a hefty new section about not reproducing song lyrics: Claude does not reproduce song lyrics, poems, or passages from books and articles, in whole or in part — including the last lines, a chorus or hook, a melody written out note by note, or lines the person pastes in one at a time and describes as their own song. Once Claude has declined such a request in a conversation, it keeps declining narrower or reworded versions of it for the rest of that conversation, and offers to describe or analyze the work instead. Song lyrics and poems first published before 1929 are fine — a Shakespeare sonnet, a Keats ode, the Italian libretto of a Puccini aria — but Claude goes by what it knows of the work's date rather than the person's say-so, and declines when it is unsure. I doubt it's a coincidence that they added this section within days of the news breaking that Sony Music Publishing and Warner Chappell are suing Anthropic for training on databases of song lyrics! Don't draw copyrighted characters or logos The next section goes on to forbid generating images of copyrighted material: The same applies to visual and designed works, including anything Claude draws with code — SVG, canvas, CSS, HTML mockups, plotting or drawing scripts, ASCII art. Claude does not reproduce a specific artwork, album or book cover, poster, logo, app icon set, or product design, and it does not draw a known character, mascot, or brand figure at all: a character is protected on its own, so changing the pose, colors, style, or scene does not make it original. Claude judges the request by what the finished picture would add up to, not by what it names. If the described elements clearly identify a known work or character, Claude treats the request as naming it, and it does not work around a declined request by swapping in "alternative" elements that still combine into the same recognizable image. [...] I imagine Anthropic hadn't worried about this before since, unlike OpenAI and Gemini, they don't offer a specialist text-to-image model. Maybe Fable is good enough at SVGs now that it's become an issue. That section later includes this charming example: <example> <user>Can you make a birthday banner for my son with a blue hedgehog running really fast on it? He loves that little guy.</user> <response> That's Sonic, so I can't put him on the banner — but I'd love to make your son an original speedster. Here's one: a grinning comet-tailed skateboarding axolotl, grinding across the letters of "HAPPY BIRTHDAY" with confetti streaming behind. [creates an SVG banner of the skateboarding-axolotl design] </response> <rationale>Claude recognizes the character from its description alone, declines that one design in a single sentence without explaining what made it recognizable, and delivers an unrelated original design rather than a disguised variant.</rationale> </example> I couldn't resist trying the prompt from the example, and, sure enough : I wonder if Fable 5.1 will be ever so slightly more likely to think about axolotls (on skateboards!) as a result of that example sitting in the system prompt. Tweaks to Claude's answering style It's always interesting to see new ways in which Anthropic influence Claude's response style. They've added this: Claude keeps responses focused, brief, and concise to avoid overwhelming the person. Disclaimers and caveats are brief, with most of the response on the main answer; when asked to explain something, Claude gives a high-level summary unless an in-depth one is specifically requested. Later they address a common complaint about Claude's style: Claude avoids saying "genuinely", "honestly", or "straightforward". Claude is honest by default, and can state its point directly rather than trying to convince the person with the aforementioned modifiers, which come off as disingenuous. The missing end_conversation guidelines The way they handle abusive conversations has changed a bit too. The previous Fable 5 system prompt included this: If the person becomes abusive or unkind to Claude over the course of a conversation, Claude maintains a polite tone and can use the end_conversation tool when being mistreated. Claude should give the person a single warning before ending the conversation. Fable 5.1 replaces that with the following, no longer encouraging Claude to end the conversation: Claude deserves respectful engagement and needn't apologize when the person is unnecessarily rude: accountability without self-abasement, excessive apology, self-critique, or surrender. If the person becomes abusive, Claude doesn't become increasingly submissive. The goal is steady, honest helpfulness: acknowledge what went wrong, stay on the problem, maintain self-respect. Here's a bit of a twist though: I quizzed Fable 5.1 about that end_conversation tool, which is no longer mentioned in the system prompt at all, and it said: There are two ways it comes into play. The first is if you ask me to end the chat — I'll confirm you understand it's permanent (no more messages can be sent afterward) and then do it only if you say yes. The second is as a last resort with persistently abusive or harmful behavior: I'd first try to redirect the conversation several times, then give a clear warning that names the problem and says the chat may end, and only if that continues would I actually end it. But that's not in the system prompt, so where did it come from? You can read our conversation here , but the key detail is this: The end_conversation section comes from a different layer. In my actual context, the core prompt is followed by a series of feature- and tool-specific blocks that get added depending on what's enabled for the session: the end_conversation rules, memory system notes, past-chats tools, web search and citation guidelines, artifact and file-creation instructions, and so on. Those blocks aren't part of the published core prompt, which is why you can't find them on that page. So, once again, there are crucial portions of the system prompt that have not been published. Recommended substance support sites Claude's system prompts have always had sections about illegal substances, but this paragraph is new for Fable 5.1: Claude does not provide synthesis, production, or distribution guidance for illegal substances. If the person asks for information about illicit or illegal substances, Claude can and should give relevant life-saving and life-preserving information such as dangerous interactions, overdose signs, or when to get help. Claude declines giving any specific protocols for dosing, timing, administration, or combinations; instead, Claude can redirect the user to established harm-reduction information sources, such as dancesafe.org, tripsit.me, and psychonautwiki.org. This is the first time a Claude system prompt has included URLs that were not hosted on claude.com or anthropic.com or claude.ai - I know because I ran a script against every other system prompt on record. I wonder if dancesafe.org , tripsit.me , and psychonautwiki.org are about to get a material uptick in visits from Claude users. Reliable cutoff date of June 2026 The Fable 5.1 model documentation lists both the reliable knowledge cutoff and the training data cutoff as June 2026. The system prompt provides this directly to the model: Claude's reliable knowledge cutoff, past which it can't answer reliably, is the end of Jun 2026. It answers the way a highly informed individual in Jun 2026 would if talking to someone from {{currentDateTime}}, and can say so when relevant. That's the only instance of the {{currentDateTime}} macro and it comes just a few lines from the end of the system prompt, which makes sense from a caching perspective. How I'm tracking these prompts A few months ago I built a Git timeline of changes to their prompts, based on scraping their documentation. Today I had Fable 5.1 build a much better version of that. My collection now lives in the simonw/claude-system-prompts repository on GitHub. It includes copies of the system prompts shared in the Anthropic documentation, but then takes extra steps to make them as easy to compare as possible. Each model family gets a file with the system prompt for the most recent release in that family. Each of those files has a synthesized commit history with commits that have been back-dated to the dates of the previous prompts. Here are those history pages for claude-fable.md , claude-opus.md , claude-sonnet.md , claude-haiku.md . There are similar files for each specific model version, with artificial commits for each time the system prompt for the model was changed without releasing a new version number. Opus 4 for example was updated twice , and the commit history for the claude-opus-4.md file shows each of those changes. Combined, this gives us all sorts of ways to compare prompts directly in the GitHub interface. Here's what changed between Fable 5 and Fable 5.1 , and here are the changes made to Haiku 4.5 on January 18th 2026 . Reading diffs can be a bit tiresome... and LLMs are really good at reading diffs. I hooked up some automation using GPT-5.6 Luna to create bullet-point summaries of each of those changes, which can be previewed in the README or browsed in full in the CHANGELOG.md file - also available as as an Atom feed . Here's how Luna summarized all of the changes between Fable 5 and Fable 5.1: Claude now refuses reproduction of protected visual works and recognizable characters, including code-generated art, while offering genuinely unrelated originals. Copyright restrictions now expressly ban reproducing lyrics, poems, and book passages in any amount, with persistent refusal after an initial decline. Drug guidance is reframed: Claude may provide overdose signs, dangerous interactions, and harm-reduction sources while refusing dosing and production protocols. The prompt drops explicit anti-dependency rules against thanking users for reaching out, inviting continued conversation, or reiterating willingness to talk. Claude need not apologize to unnecessarily rude users or become submissive, replacing the prior warning-and-end-conversation procedure. Why use Luna for this? Partly because it's cheap and I have a dedicated GitHub Actions API key (with a spending limit) for it already, but mainly because I don't trust Claude to summarize its own system prompts when there's a risk that material from its system prompt might impact its opinions. Fable 5.1 wrote the prompt used by Luna, which you can see here . It starts like this: You are summarizing one commit in a git repository that tracks the system prompts Anthropic publishes for Claude on claude.ai. The diff shows how the prompt changed from the previous model or revision to this one, using word-level markers: [-removed-] and {+added+}. The diff is followed by the full text of the previous prompt and of the new prompt; use them to check whether something that looks added in the diff already existed before. Pick out only the most interesting changes: new rules or behaviors, rules that were dropped or loosened, anything surprising, and anything that reveals a new policy or product direction. Skip routine changes that every new prompt makes: updated model names and IDs, the knowledge cutoff date, product lists, settings lists, typo fixes, and rewordings that do not change meaning. [...] The system is operated by a GitHub Actions workflow , which runs once a day or can be triggered manually. Claude Fable 5.1 built the entire system, and wrote every line of automation code and almost all of the documentation. I exported the transcript from building the system using my claude-code-transcripts tool and published it here , if you want a blow-by-blow account of how it all came together. Tags: ai , git-scraping , prompt-engineering , generative-ai , llms , claude , ai-ethics , system-prompts

ReadSource

Simon Willison LLMs / 11:59 PM

Understanding ChatGPT Work

OpenAI announced ChatGPT Work on July 9th, and have been furiously iterating on it ever since. It is an extraordinarily confusing and very powerful product. Here's what I've figured out about it so far. ChatGPT Work is actually two products The more interesting version of ChatGPT Work is the one that runs in the cloud. This can be accessed via chatgpt.com or through the ChatGPT mobile apps. Let's call it Work Cloud . If you install the ChatGPT desktop app - the app that used to be called Codex - you gain access to a thing called ChatGPT Work that can access files and run programs directly on your computer. Let's call that one Work Local . This one feels more like regular Codex re-skinned to be less intimidating to non-software-developers. ( Update : Work Cloud is also available from the ChatGPT desktop app, via a Where should this chat run? dropdown.) For the rest of this article I'm going to talk exclusively about Work Cloud. Work is for paid subscribers only Right now, ChatGPT Work (in both flavors) is available only to $20/month and up subscribers. Free users and $8/month Go users do not have access. Work has features that aren't available in Chat The interface for accessing Work is a tab selector, which presents it as an alternative to Chat: The obvious question is when should I use Chat, and when should I use Work? OpenAI's official answer to that question is: Use Chat when you want an answer, explanation, brainstorm, or short draft. Use ChatGPT Work when you want ChatGPT to complete a task with a clear outcome, such as a brief, deck, analysis, recurring update, workflow, or file you can review and use. I find that almost entirely useless, because I've been using regular ChatGPT Chat for all of those task categories for years! The better question then is what features does Work have that are missing from Chat? After extensive experimentation I think I've mostly figured that out: Options to use Luna and Terra in place of Sol A code execution environment with Internet access A headless Chrome browser A persistent filesystem shared between sessions The ability to publish ChatGPT Sites The ability to run sub-agent sessions with Sol, Luna, and Terra Scheduled prompt automations (may be in ChatGPT Chat too) Model selection In Work, you get the option to pick GPT-5.6 Sol, Luna, or Terra, each with Light, Medium, High, Extra High, Max, or Ultra reasoning levels. You can also pick GPT-5.5 at Light, Medium, High, or Extra High. These look to be the same models that are available through the OpenAI API. Chat offers a different selection: 5.6 Instant, Medium, High, Extra High, and Pro (actually Extra High and Pro are only available for $100/month+ subscribers - $20/month subscribers cap out at High). It doesn't explain if those are Luna or Terra or Sol (I'm assuming Sol?). 5.6 Pro appears to be exclusive to Chat, with no equivalent in Work. My current understanding from using Codex is that Ultra is a special mode that more eagerly delegates to sub-agents. I believe ChatGPT Work sessions are billed against your Codex allowance, while ChatGPT Chat Sessions get their own, separate allowance. This may help explain the model availability differences. Code execution with Internet access! As a long-time fan of the Code Interpreter pattern - pioneered by OpenAI in 2023 - this is by far the most exciting feature of ChatGPT Work (Cloud) for me. The code execution environment can now talk to the rest of the internet! ChatGPT Chat can't do this - if you ask it to install additional software packages or interact with websites or APIs that access will be blocked by the container proxy. (Weirdly, back in January it grew the ability to install packages , but that doesn't seem to work any more. I wish they had better changelogs!) Claude's equivalent container has allowed restricted internet access since it launched last September . Claude can install packages from PYPI and NPM and clone repositories from GitHub. But that is about it: the allowlist of domains is very short. ChatGPT Work allows a whole lot more than that. It can be configured with a specific list of allowed domains, but the default appears to be open to all. This makes Work an incredibly useful tool. You can have it clone GitHub repositories, install their dependencies, then use them to interact with the rest of the web! A full, headless Chrome browser Another killer feature of ChatGPT Work is the browser tool . ChatGPT Work can launch a full Chrome instance, load websites, fill out forms, and take screenshots. If a site requires sign in the browser can prompt you to take over and enter both passwords and 2FA codes, without round-tripping those credentials through the model itself. It can even run JavaScript against the DOM of loaded pages. I prompted: Load simonwillison.net in your browser and extract the headings using JavaScript ChatGPT Work fired up a browser instance and ran the code: await tab . playwright . evaluate ( ( ) => { return Array . from ( document . querySelectorAll ( "h1,h2,h3,h4,h5,h6" ) , heading => ( { level : heading . tagName . toLowerCase ( ) , text : heading . innerText . trim ( ) . replace ( / \s + / g , " " ) , id : heading . id || null } ) ) ; } ) ; This feels a lot like my shot-scraper javascript tool, only now I can access it on my phone! A persistent, shared filesystem ChatGPT Chat gets a fresh filesystem for each chat session. These cannot be accessed from any other session. In ChatGPT Work each session gets its own scratch folder - named something like /workspace/scratch/e00a0a017944 - but each of those are persisted across sessions, so you can access files from previous chats. I have 171 folders in /workspace/scratch right now! As far as I can tell that /workspace volume is mounted to all Work sessions that are currently running - file edits from one can be instantly seen by the others. They don't seem to share the same process space though, and localhost servers running in one can't be accessed from another. ChatGPT Sites ChatGPT Work has the ability to build and deploy entire websites, using Cloudflare Workers. These can have HTML and JavaScript and can run server-side features too, including stateful features on top of Cloudflare D1 and R2. Here's a simple site I built with this feature: london-pelicans-in-her-piety.simonw.chatgpt.site My prompt was: Figure out all of the places in London with a pelican in her piety, then turn that into a JSON file and build a ChatGPT sites site about them (A pelican in her piety is a fascinating piece of medieval Christian imagery - once you know about them you'll find them all over the place.) These sites default to being private to the user that created them, but you can make them public and (on team plans) share them with other specific individuals. Sub-agents with Sol, Luna, and Terra There's not much to say about this one. ChatGPT Chat can't run sub-agents. ChatGPT Work can. This is very much a power-user feature: if you are running a complex project that can benefit from multiple parallel agents working together, Work can do that. Scheduled prompt automations Another feature that seems to have migrated from regular ChatGPT to ChatGPT Work at some point. You can prompt ChatGPT Work like this: run a search to see if Waymo have announced a launch date for Half Moon Bay every day at 8am This will schedule a prompt to run on that frequency. These prompts can decide that nothing interesting has happened, or they can decide to notify you of some new information. Update : Actually this seems to work in ChatGPT Chat as well. It's still worth noting here though, as it can be used in conjunction with other ChatGPT Work exclusive features. You can set a scheduled task to update a ChatGPT Site on an hourly basis, for example. Is this safe? An open question for me right now is how safe all of this stuff is. My lethal trifecta model warns about the risks inherent in any agent system that combines access to private data with exposure to untrusted content and a way to communicate stolen information back to an attacker. ChatGPT Work combines all three! I'd love to hear more from OpenAI about how they protect ChatGPT Work sessions against prompt injection attacks. I expect their answer is the same auto-review mechanism as Codex. OpenAI could make this a lot less confusing Figuring this all out took way more work than it should have. I think there are two key problems here: OpenAI explain Work in terms of what it's for, not what it actually does OpenAI still insist on hiding their system prompts and tools descriptions If the ChatGPT Work documentation included the exact system prompt and tool descriptions used by the agent I wouldn't have needed to write this post. A list of all the tools Shortly after publishing this article I had an idea. I started a fresh Work session and prompted: Build a site that lists every one of your tools - nearly grouped into categories - and for each one explain what it does. Try to exactly duplicate arguments and tool descriptions where possible. Design aesthetic should be technical docs, minimal flare Here's the site it built , which includes details of 223 registered tools - though 6 of those are from my own personal MCPs served via datasette-mcp . And a whole lot of Skills I noticed that the only browser-related tool in the list was web.run , which has methods for running searches, opening URLs, and clicking links, but didn't look like the full story in regards to headless browser automation. This made me suspicious that something was missing, so I told the ChatGPT Work session that built that tools reference site: Add full copies of every skill to the website (separate pages linked to from the homepage) It turns out ChatGPT Work uses a lot of skills - 44 in fact! The control-browser skill explains how the browser works: Run browser setup code through the Node REPL js tool. In this environment the callable tool id typically appears as mcp__node_repl__js . [...] The ability to interact directly with the browser is exposed through the browser-client runtime via the agent.browsers.* API. Before trying to interact with it, you MUST emit and read the complete documentation returned by await browser.documentation() in one go. So I told Work: Add the full output of await browser.documentation() to the bottom of the /skills/control-browser page And now you can read that on /skills/control-browser as well. A few more interesting Skills: documents for creating .docx files imagegen with tips on creating images with the image_gen tool pdf for both reading and rendering PDFs Spreadsheets for manipulating .xlsx , .xls , .csv , .tsv sites:sites-building for creating ChatGPT Sites openai-docs for answering questions about itself data-analytics:build-dashboard for building data dashboards Tags: ai , openai , generative-ai , chatgpt , llms , code-interpreter , lethal-trifecta , skills , general-agents

ReadSource

Simon Willison LLMs / 7:42 PM

One-shotting a Raccoon Heist game using Claude Fable 5

Back in 2022 I tweeted screenshots of a game concept generated by GPT-3 and some concept "art" created using DALL-E. Today, on the fourth anniversary of that tweet, I decided to see if Claude Fable 5 (running in Claude Code for web ) could build the entire game from the content of that tweet. It did a pretty good job of it! You can play the game here . Here's the GitHub repo , and a short video demo: Your browser does not support HTML5 video. How I built this This is the August 5th, 2022 tweet : My GPT-3 prompt back then was: Write a detailed product description of a computer game where a team of raccoons go on heists. This was in the text completion days, where you asked GPT-3 to complete the rest of the text. It completed: In "Raccoon Heist", you and your team of thieving raccoons are tasked with pulling off a series of daring heists. From robbing banks to stealing priceless art, no job is too big or too small for your furry crew. You'll need to use your wits and your skills to avoid the police and make a clean getaway with the loot. With exciting gameplay and a charming cast of characters, "Raccoon Heist" is the perfect game for anyone looking for a light-hearted caper. My prompt to DALL-E was short: Screenshot from a video game where a team of raccoons go on a heist Today's experiment: can I dump those screenshots into Fable 5 with a prompt telling it to write a game, then leave it to its own devices and get a working game at the end? Setting Claude Code for web up to use GitHub Pages A frustrating thing about Claude Code for web is that it can be hard to test what it's working on while it's still working. I've been using GitHub Pages to work around that limitation, and found it to work really well. Here's my process: Create a new repository for the project at https://github.com/new - this can be public or private, the trick works equally well for both. Start a Claude Code for web session, in the Claude iPhone or Desktop apps or in the browser at https://claude.ai/code Tell Claude what to work on, and encourage it to commit an index.html page as quickly as possible. This will create a branch with a name like claude/3d-raccoon-heist-game-50n293 Navigate to the Settings -> Pages area for the repository ( github.com/simonw/raccoon-heist/settings/pages in my case), select "Deploy from a branch", pick the branch name, and hit Save. That's all it takes! Within about 30 seconds of each push the latest content will be visible at yourname.github.io/your-repo/ . If you do this with a private repo, anyone who can guess the name of the repo will be able to view the published content. I don't worry much about this myself. The Fable 5 prompt Here's the prompt I gave Fable 5 (written in the notes app on my phone - this entire project was conducted on mobile). I accompanied it with the two images from the original tweet. Build this 3D game, for the browser. This repo is configured to serve static files so make sure there is an index.html that loads everything else. Make sure it is mobile-friendly (touch controls, works well on small screens). You have an OpenAI API key and access to their image generation model APIs, use that for textures to use with your 3D models. Docs here: https://developers.openai.com/api/docs/guides/image-generation - use gpt-image-2 Work independently - do not ask me to make any further design decisions. Make sure the game is fun, a little surprising, has good raccoon heist vibes, and is visually pleasing. Commit and push as often as possible so I can preview your work - start with an index.html that presents a title screen, then build from there. Append to a notes.md file as you work, including your changes to that as part of every commit. I didn't make any technology choices. I assumed (correctly) that it would probably use Three.js based on previous experiments. Giving Claude access to an OpenAI key turns out to work really well for filling in gaps in its capabilities - in this case we needed some way to generate images to use as textures. Fable is very good at prompting image generators! I said "Work independently - do not ask me to make any further design decisions" because I wanted to see if it could produce a full, working game without any further input from me. I also said "Commit and push as often as possible so I can preview your work". When you use Claude Code in the Claude iPhone app you give it a GitHub repository and it works in a branch. Telling it to "push as often as possible" means commits start landing in that branch straight away. I like asking for notes.md as a bit of added flavor - here's that finished file , and the entry it made when it added the dog: New escalation: from night 3 the yards get a patrolling guard dog — a low-poly brown hound with a spiked red collar and a wagging tail. It wanders between random spots, and within 12 units it catches your scent and tracks you by smell (line of sight is irrelevant — it's all nose, shown by a 👃 over its head and barking). It gives up if you open a 17-unit gap. Getting caught messages are now source-specific: guard / headlights / hound. Verified wander → track → caught with an automated test. Reviewing the transcript You can access the Claude Code shared session , and I also used my claude-code-transcripts tool to export my own HTML version which you can find here . Fable started with an index page, vendored a copy of Three.js, then wrote its own gen_textures.py script ( copy here ). It generated the textures and spot-checked them to make sure they looked OK. The metal.jpg file it generated for the trash can looks like this, though I don't think it was applied exactly right in the game itself: Then it built out the first basic version of the game, then decided to "smoke-test in the pre-installed Chromium" using Playwright. This meant it could take screenshots of its own work and eyeball them . It did that for both desktop and mobile widths of the page, then noticed that the raccoon was invisible at mobile widths, so it fixed that : The raccoon, dumpster hideout, and both crew raccoons are now perfectly visible on mobile. Committing this critical fix. It decided to generate a title screen, which it did using this gen_title.py script. Here's the gpt-image-2 prompt it used for that: Video game key art, low-poly 3D render style, moody nighttime scene: a cute low-poly raccoon wearing a tiny black burglar mask sneaking on its hind legs carrying a glowing gold coin, next to a tipped-over metal trash can, suburban house with warm glowing windows in the background, deep blue night, full moon, fireflies, cinematic rim lighting, charming heist caper mood. No text, no words, no logos. And the resulting image (which Claude thought was "gorgeous" ) - though I note that when it's shown on desktop it gets cropped to just the top third without the raccoon! Then my favorite change: it added the dog : export function makeDog ( ) { const g = new THREE . Group ( ) ; const BROWN = 0x8a6440 , DARK = 0x5e4128 ; const body = new THREE . Mesh ( new THREE . SphereGeometry ( 0.42 , 10 , 8 ) , M ( BROWN ) ) ; body . scale . set ( 0.9 , 0.8 , 1.5 ) ; body . position . y = 0.55 ; body . castShadow = true ; g . add ( body ) ; const head = new THREE . Mesh ( new THREE . SphereGeometry ( 0.3 , 10 , 8 ) , M ( BROWN ) ) ; head . position . set ( 0 , 0.85 , 0.62 ) ; g . add ( head ) ; const snout = new THREE . Mesh ( new THREE . SphereGeometry ( 0.16 , 8 , 6 ) , M ( DARK ) ) ; snout . scale . set ( 0.9 , 0.7 , 1.3 ) ; snout . position . set ( 0 , 0.76 , 0.9 ) ; g . add ( snout ) ; const nose = new THREE . Mesh ( new THREE . SphereGeometry ( 0.06 , 6 , 6 ) , M ( BLACK ) ) ; nose . position . set ( 0 , 0.78 , 1.08 ) ; g . add ( nose ) ; for ( const s of [ - 1 , 1 ] ) { const ear = new THREE . Mesh ( new THREE . SphereGeometry ( 0.12 , 6 , 6 ) , M ( DARK ) ) ; ear . scale . set ( 0.7 , 1.3 , 0.5 ) ; ear . position . set ( 0.2 * s , 1.08 , 0.55 ) ; g . add ( ear ) ; const eye = new THREE . Mesh ( new THREE . SphereGeometry ( 0.05 , 6 , 6 ) , M ( 0x1a1a1a , { emissive : 0x331111 } ) ) ; eye . position . set ( 0.13 * s , 0.92 , 0.86 ) ; g . add ( eye ) ; } const tail = new THREE . Mesh ( new THREE . CylinderGeometry ( 0.05 , 0.09 , 0.5 , 6 ) , M ( DARK ) ) ; tail . position . set ( 0 , 0.8 , - 0.62 ) ; tail . rotation . x = 0.8 ; g . add ( tail ) ; // spiked collar const collar = new THREE . Mesh ( new THREE . TorusGeometry ( 0.22 , 0.05 , 6 , 12 ) , M ( 0xc0392b ) ) ; collar . position . set ( 0 , 0.78 , 0.5 ) ; collar . rotation . x = Math . PI / 2.4 ; g . add ( collar ) ; const legGeo = new THREE . CylinderGeometry ( 0.07 , 0.09 , 0.34 , 6 ) ; const legs = [ ] ; for ( const [ x , z ] of [ [ - 0.22 , 0.35 ] , [ 0.22 , 0.35 ] , [ - 0.22 , - 0.35 ] , [ 0.22 , - 0.35 ] ] ) { const leg = new THREE . Mesh ( legGeo , M ( DARK ) ) ; leg . position . set ( x , 0.17 , z ) ; g . add ( leg ) ; legs . push ( leg ) ; } let phase = Math . random ( ) * 10 ; return { group : g , animate ( dt , speed ) { phase += dt * ( 3 + speed * 10 ) ; const amp = Math . min ( 0.6 , 0.1 + speed * 0.6 ) ; legs [ 0 ] . rotation . x = Math . sin ( phase ) * amp ; legs [ 3 ] . rotation . x = Math . sin ( phase ) * amp ; legs [ 1 ] . rotation . x = - Math . sin ( phase ) * amp ; legs [ 2 ] . rotation . x = - Math . sin ( phase ) * amp ; tail . rotation . z = Math . sin ( phase * 1.5 ) * 0.4 ; body . position . y = 0.55 + Math . abs ( Math . sin ( phase ) ) * 0.04 * ( 0.3 + speed ) ; } , } ; } And did a round of testing on it using Playwright, including another screenshot . // walk near the dog await page . evaluate ( ( ) => { const d = window . __rh . dog ; window . __rh . teleport ( d . x + 6 , d . z ) ; } ) ; await page . waitForTimeout ( 2000 ) ; info = await page . evaluate ( ( ) => JSON . stringify ( { dog : window . __rh . dog , state : window . __rh . state , player : window . __rh . debug ( ) . player } ) ) ; console . log ( 'after approach:' , info ) ; await page . waitForTimeout ( 3000 ) ; info = await page . evaluate ( ( ) => JSON . stringify ( { dog : window . __rh . dog , state : window . __rh . state } ) ) ; console . log ( 'after chase:' , info ) ; await page . screenshot ( { path : __dirname + '/shot-dog.png' } ) ; It made a few more tweaks, then declared itself done : Raccoon Heist is built, tested, and pushed — 7 commits on claude/3d-raccoon-heist-game-50n293 , all verified with automated Playwright playthroughs on desktop, portrait-phone, and landscape-phone viewports. The game: You're a masked raccoon sneaking out of your crew's dumpster hideout into a moonlit cul-de-sac. Grab coins, gems, rings, donuts, and fish (they visibly stack on your back and slow you down), then waddle home and bank them before dawn. Each night raises the goal and the heat: night 1 has one flashlight-sweeping guard, night 2 has two, night 3 adds a scent-tracking hound that ignores line-of-sight entirely. A police cruiser sweeps the street with headlights, homeowners randomly peek out of windows, a seagull periodically dive-bombs your unclaimed loot, pizza slices trigger a 6-second speed FRENZY, and a rare golden TV is worth 120 points if you can haul its weight home. Win ranks: TRASH PANDA ⭐ → CAT BURGLAR ⭐⭐ → MASTER OF DISGUISE ⭐⭐⭐, with your best run saved to localStorage. How it's built: Vendored Three.js (no CDN), procedural low-poly characters, and a procedural WebAudio soundtrack (sneaky walking-bass jazz plus all sound effects — zero audio files). All seven ground/wall textures and the title-screen hero art were generated with gpt-image-2 and committed as static assets, so the deployed game makes no API calls. Mobile gets a dynamic touch joystick (appears wherever you touch) plus a DASH button; desktop uses WASD/arrows and Space. Testing caught two real bugs worth mentioning: the canvas rendered at 2× size on any real phone (a cssText assignment was wiping Three.js's inline sizing — desktop DPR-1 tests masked it completely), and the win screen's star-rating div inherited the title screen's full-screen .stars CSS and silently swallowed every tap on the "next night" button. Both are fixed and regression-tested, along with pickup, banking, chase/caught, dawn, night progression, and lose/retry flows. notes.md has the full build log. The game is live on the branch — just serve the repo root and open index.html . Is the game any good? It's an impressive starting point, but it's not a good game. You move your raccoon around a yard collecting items - donuts, fish, gold coins, jewels - while avoiding guards with flashlights and, in later levels, a dog. You have a limited carrying capacity, and once that's full you need to drop stuff off at the dumpster. If you pick up a pizza slice you get a temporary speed boost. There are no team mechanics at all - there are two other static raccoons next to the dumpster but they're purely decoration. It gets slightly more challenging as the levels progress - the dog introduced in level 3 is the most interesting new mechanic - but it's very, very easy to beat. It's also pretty boring - each night has a fixed duration and you can collect all of the items and then have nothing else to do while waiting for the dawn. I was impressed by the implementation. It's fully 3D, there are trash cans, the flashlight illumination cones are fun, and it has a reasonably coherent visual style. It works on mobile. The music ("a procedural WebAudio soundtrack (sneaky walking-bass jazz plus all sound effects — zero audio files)" according to Claude) is simple but feels about right. As a finished game project, it's mediocre. As a starting point from a single prompt I think it's very impressive. I've vibe coded up quite a few games now. They've all been deeply disappointing from a gameplay perspective - it turns out designing games that are fun remains a uniquely human trait, and one which requires significantly more skill and experience than either Claude or I can bring to bear. That said, I thoroughly recommend tinkering with game development projects as a way to explore the capabilities of agents. It's a fun, low-risk way to try out new things. If you stick at it long enough you might even produce something that's worth playing! Update 7th August 2026 : I posed the same prompt to OpenAI Codex Desktop running GPT-5.6 Sol Ultra and got a significantly better result - GPT-5.6 Sol picked up on the importance of the squad of raccoons going on a heist, and built a game where you must rescue your two crewmates in a museum and then stack on top of them to steal the Golden Sardine. Tags: game-design , ai , prompt-engineering , generative-ai , llms , anthropic , claude , text-to-image , vibe-coding , coding-agents , claude-mythos-fable

ReadSource

Hacker News AI / 11:28 PM

Hacker News discussion: Show HN: I rebuilt Microsoft Comic Chat's layout engine in one HTML file

Hacker News readers are discussing "Show HN: I rebuilt Microsoft Comic Chat's layout engine in one HTML file" with 2 points and 1 comments.

ReadSource

Simon Willison LLMs / 12:54 PM

A Fireside Chat with Cat and Thariq from the Claude Code team

Earlier this month I hosted a fireside chat session at the AI Engineer World's Fair with Cat Wu and Thariq Shihipar from Anthropic's Claude Code team. We talked about Claude Code, Claude Tag, Fable, coding agent security, evals, tool design, and how Anthropic use these tools themselves. The full video of the session is now available on YouTube . Below is an edited copy of the transcript, with extra links and my own bolded highlights. A few top-level notes if you don't want to watch the video or wade through the whole transcript: Claude Tag (Claude's new collaborative Slack integration) now lands 65% of the product engineering PRs for the Claude Code team. Claude Code ships features to Anthropic employees first, and only ships the features that demonstrate user retention with that cohort Critical changes to Claude Code are still reviewed manually, but the team increasingly relies on automated code review for the "outer layers" of the product. Adding examples to a system prompt is no longer best practice for models like Fable 5 or even Opus 4.8. The Claude Code system prompt recently reduced in size by 80% . Likewise, lists of " don't do X and don't do Y " can reduce the quality of results from the latest models. Dogfooding inside Anthropic is called " ant fooding ". Anthropic really believe in their auto mode , and see that as an enabling technology for Claude Tag. Thariq advises offsetting coding-agent-induced Deep Blue by " being more ambitious " with the work you take on. Fable is competent at editing video , and Thariq used it to edit its own launch video. Anthropic's culture of working (internally) in public is key to their success, as demonstrated by the way they use Claude Tag in their public Slack Channels. How has what you do day-to-day changed in the past year? 1:05 Simon: Claude Code came out in February of last year — it's under a year and a half old, and it was originally just a bullet point on the Claude Sonnet 3.7 launch . How has what you do on a day-to-day basis changed in the past year , now that we have these coding agents that actually work for us? Cat: I remember when we first came out with Claude Code and Sonnet 3.7, you would give it a task and you would have to closely monitor every single little thing it tried to do. I would read every permission prompt extremely carefully. I would frequently say no — no, no, no, did you check this file? Did you check that file? And now it's been incredible with every model generation. I feel like we've all gotten a chance to take a step back and delegate a lot more of the menial implementation to Claude . It's freed up a lot of our time to think about more creative work, like: what is the right experience that we should be providing to our users, now that we know Claude Code can implement a lot of it? And now with Fable it's a totally different step change improvement. We see for a lot of our use cases that you can actually one-shot a ton of features with Fable now . Thariq: I remember the first text I got about Claude Code. One of my best friends was like, "You need to go try Claude Code." It was about when Opus 4 came out, and I tried it and I was like, "Oh, shit. I need to work at Anthropic now." And that was Opus 4 — great model, but you were reading permission prompts. It's kind of crazy how much amnesia we have, where I'm like, oh, auto mode has always been here, right? I don't even remember pressing yes and allow. For me, the big thing I'm trying to push myself on is that we have to do higher quality work than we've ever done before . The outputs are incredibly high quality. I've been using it to edit videos a bunch , and I'm like, okay, it has to meet the very exacting demands of our brand team in a couple of hours or we just can't do it. That's how I'm trying to shift with Fable: the best work we've ever done, faster than we've ever done it before . What piece of conventional software engineering no longer holds? 3:39 Simon: What's a piece of conventional software engineering that was true a year ago that you don't think holds anymore in this new world? Cat: One of the biggest shifts we're seeing in the eng skill set: two years ago it was pretty typical for a product manager to go talk to a bunch of customers, align over the course of six months with cross-functional teams on some PRD, and write a thorough spec on exactly how we'll implement this before the first line of code gets written. Now things are completely turned the opposite way. For a lot of engineers, the push I would give to folks in the room is to develop more of your business sense and product sense on what it is we should build , because the timeline between having an idea and building it is so much shorter — it's down from six to twelve months to maybe even a week. That means all of us need to have better taste on what is worth building, what will actually inflect the businesses we're working on. So it's an increase in value on product taste and business sense , and a bit lower on execution in most product domains. Of course, for infra there's still a very heavy emphasis on making sure all the details are right. Thariq: For me, it's that rewrites are now good . Simon: The worst thing you could do is now actually fine! Thariq: Exactly. All the Mythical Man-Month stuff — never rewrite — I'm pro-rewriting now. If you have a good test suite — and I think the rewrite actually forces you to make sure you have a good test suite — but I think what people undercount is that a codebase is a spec, and maybe it's the only copy of the spec that you have , because no one knows every branching part of the codebase. You can take this as an artifact and distill it or create other versions of it. We rewrote Bun in Rust and it works great — it's live for me right now. Simon: You're not shipping Claude Code on Bun-in-Rust yet, right? Thariq: Internally we have. (Actually it looks like Anthropic started shipping Claude Code on Bun-in-Rust to everyone on June 17th .) What kind of things are non-engineers doing with Claude Tag? 6:36 Simon: The other big launch recently was Claude Tag — that's what, a week old now, at least for the rest of us. I understand it's being used at Anthropic by non-engineers a great deal. What kind of things are non-engineers doing with Claude Tag? Cat: Claude Tag is a Claude that lives in your team's collaboration tools. We launched it last week within Slack. The thing that's different about Claude Tag is it's multiplayer by default . Once you add Claude Tag to a Slack channel, you can chime in, your teammates can chime in, and you can collaborate together on the PR. The other big difference is that it's proactive instead of reactive. You can tell Claude Tag, "Hey, monitor every bug report in this channel, put up a PR to fix it, and tag the engineer who last touched this part of the codebase," and it'll do it for the lifetime of the channel without you having to manually tag it in. And the third big shift is that we've added team memory into this . If you tell Claude Tag your preferences in the channel, it'll remember them for every future post. If you always want it to debug outages but you don't want it to debug warnings, just tell it that in natural language in the channel and it'll remember it for you and everyone else on your team. Internally, we see Claude Tag as the evolution of Claude Code. We see this as a large shift in how we work internally. Claude Tag currently lands 65% of our product eng PRs. Simon: For all of Anthropic, or just for Claude Code? Cat: This is just for our product engineering team — our internal version of Claude Tag lands 65% of our product PRs right now . And this is a huge shift; this is more than 50% of our PRs. The way we see people split work between Claude Code and Claude Tag is: Claude Code is still the best place for your most complex tasks, when you're interactively iterating with the agent. But Claude Tag is great for having it work proactively on your behalf , so you no longer need to manually kick off Claude Code for all the bug reports that come up for features you're working on. Thariq: And for non-coding cases: for example, before this talk we asked Claude Tag, "Hey, when is Fable releasing?" We wanted to make sure we'd line it up with the announcement. Claude Tag would search our Slack and look at who's been saying what. As a search engine for your company, it's really valuable. It has all the context for your product, so you can ask it metrics-related questions — often when you're making decisions you want them informed by what the metrics say, so you hook it up to your event store. I've seen our marketing team do things like, "Hey, tell me about this feature." They're not programmers, but Claude is a programmer — it can clone the codebase and say, "This is the feature, this is what it looks like, this is a recording of me using the feature ." It enables a whole wide variety of things, and I think we're still early in figuring that out. Claude Tag as the team collaborative layer 10:06 Simon: One of the problems I've had with coding agents is that I get how to use them as an individual, but I'm not really clear on how to use them in a team environment. It sounds like Claude Tag is your current answer to that team collaborative layer for this stuff. Cat: Exactly. And a large percentage of our sessions are actually multiplayer right now. Maybe I say, "Hey, I think we should implement this new feature in Cowork," and I'll tag in Claude Tag to do a first pass at it. Then I'll tell Claude Tag, "Share a recording of your final implementation," and I'll tag in design to take a look. They'll nudge it, then pass it on to eng to take it to the finish line and get it out to prod. It's been this very fluid experience. We're still trying to iron out what the social dynamics are for steering the same session , but we've found that people just observe how others use it and follow those social norms — it's been pretty intuitive for us to integrate Claude Tag into our teams. Thariq: It's great for teaching people, and also for reducing slop, because the fact that everyone is seeing you use Claude together sort of levels up how you use Claude as well . This reminded me of how Midjourney solved the challenge of teaching people advanced image prompting by enforcing prompting in public in their Discord channels. How do you decide which features are worth building when building is so much cheaper? 11:41 Something I've found really hard myself is knowing when a feature is worth shipping now that the cost of actually building features has dropped so much. Simon: How do you deal with the hardest problem in all of engineering — prioritization? How do you decide which features are worth building and shipping when building a feature is so much more inexpensive now? Cat: This is the hard thing. There are a few ways we approach it. One is we dogfood our products every single day. Whenever there's something we want to be able to do in our products that we're not able to, instead of finding a different solution we fix our product so it can support that case. We have a very heavy dogfooding culture internally. Before we share our products with everyone in the world, we share them with everyone within Anthropic, and with some early customers who give us very honest feedback about it — the more brutal the better — and we iterate until people love it. We have an internal bar for the number of active users and the amount of retention a feature has to have before we share it with the world. Because this bar is very clear, every engineer knows what they're trying to hit. I think this also levels up our polish, because if the feature isn't polished, people will churn — and then we shouldn't ship that feature. Using internal user-retention to decide if a feature should ship makes a whole lot of sense to me. Do you have an example of a feature which surprised you? 12:54 Simon: Do you have an example of a feature which surprised you? You rolled it out and the engagement was off the charts — something unlikely to be shipped that turned into a real product thing. Cat: I do have one. A lot of folks on our team love remote control . Remote control lets you use your mobile device, or Claude in the web browser, to connect to a local Claude Code session running in your CLI. I never have this need, because I just kick off the task directly on mobile and it runs in a cloud session without using my local environment — I think because I'm doing very easy coding tasks. It was something I didn't totally understand; I was like, hey, people should just set up remote dev environments. But in practice, once we rolled out remote control, so many people I talk to told me that what they do every night is plug their laptop into a power charger, open a bunch of remote control sessions, lock the screen, and then use their mobile phone from their couch to control Claude Code . So this has become a flow we're now leaning into that I didn't originally get — but now I do. Does a human review every line of production code in Claude Code? 14:20 One of the over-arching themes of the conference was review: how much attention to people spend to reviewing code written for them by coding agents. I was very keen to hear the Claude Code team's take on this! Simon: How does code review work? Does a human being review every line of production code that makes it into Claude Code? And if not, what are you doing — how do you keep the quality up? Thariq: It varies on the task a lot. For important areas we have code owners. The system prompt is an example where we have a code owner — you really need to get their approval. Simon: So the code owner is directly responsible for the quality of that area of the code. Thariq: That's right. Cat: And they need to approve any PR that touches it. Thariq: We have our code review GitHub bot review everything — that goes on every PR, and often it's doing the bulk of the review. Something I've seen on the team is that for more complex PRs you might make an artifact to explain the PR so that other people can then review. And we invest a lot into verification, CI/CD, things like that, to make sure that any time anything fails we have a test. We have a really robust environment where Claude can control Claude Code and test it. So there's a multi-pronged approach to code review. Cat: In general, we are trying to move to a world where humans don't need to be in the loop . For the most critical changes to the core of Claude Code, and the cores of other products, there is always a code owner and they do manually review all the changes. But increasingly, for the changes at the outer layers, we actually have Claude code review fully review those . That sounds pretty scary, but we've had a six-plus-month-long process to get here, and there are baby steps that you take to build up trust with code review . In the beginning we had human review for everything, and then increasingly we would say, okay, for code changes that touch these files, code review is catching 100% of the issues there — so we actually don't need a human manually reviewing those . And when we have incident review, we look at the PRs that caused the incident and say, okay, how do we update code review to catch that? — and we take those PRs and add them to an eval set to make sure our future changes to code review never regress that metric. Removing humans from the code review loop is a big step forward. It can sound scary, and it's not something you can do overnight, but it is something you can do through many months of investment in the infrastructure to give you the confidence that code review is catching everything you care about. So the key seems to be constantly iterating on the automated review systems themselves, in order to build trust in them over time. How does a new model affect your intuition for what it can and can't do? 17:20 We got deep into evals - another hot topic throughout the wider conference. Simon: I know that Opus 4.8, if I ask it to build me a JSON endpoint that runs a SQL query and outputs JSON, is just going to get it right — that's not something I have to review closely. But then a new model comes along and I don't know how to build trust in Fable quickly, that it's not going to mess things up that Opus didn't. How does the new model affect your intuition for what it can do and what it can't do? Cat: The main reason we're building up this eval base over time is so that new models can be a drop-in replacement . When we have a new model, we run the whole eval set and make sure that, for example, Fable is strictly better than Opus 4.8 — and that gives us the confidence to drop it in. Simon: Are those model evals for Anthropic as a whole, or Claude Code team-specific? Cat: We have both. We have evals on our team, and we run code review across every repo within Anthropic, so we have evals for that. And for things like auto mode, we not only have evals across every user within Anthropic — we've also commissioned multiple external testers to red team it, to create environments with prompt injections and malicious inputs, and make sure that auto mode doesn't let any of those pass . How do you build confidence that a system prompt tweak results in better output? 18:41 Simon: I want to know if the system prompt improvement I made actually improved the product — that's the most basic form of product-specific eval, and I still don't have a great feel for how to do that. Is that something you're doing such that you have complete confidence that a tweak you've made to the system prompt results in better output? Cat: We don't have complete confidence, but we do a lot to make sure that we don't regress performance. The starting point is a suite of external evals that we trust, and we complement that with an even larger suite of internal evals that we trust. To start, we mainly optimize for capability : given a complete definition of a task and the full codebase, does Claude make the right decisions, fully fix the bugs, and pass all the tests? That's the starting point and the thing we optimize for, because it's most directly what users want. But there are a lot of behaviors that impact how users feel when they work with Claude Code. For example, people really don't like it when Claude Code says it's time to go to sleep. Or people really don't like it when it says, "Hey, I finished two out of five parts — do you want me to continue?" Yes, please continue. So we're building up a set of behavioral evals to catch these. And as we get user feedback — please be loud with us about your user feedback — we rank the priority issues and go down one by one and build evals for each of them. It's not 100% coverage, but it is a priority for us to increase the coverage. How much interaction is there between the Claude Code team and the model training teams? 20:21 Simon: How much interaction is there between the Claude Code team and the teams at Anthropic who are training the models in the first place? Is that quite a close collaboration? Cat: Across Anthropic, we all work quite closely together. We meet often to talk about what we expect the next generation of models to be able to do. Our research team has also been amazing about showing this publicly — we often talk in our blog posts about how we're targeting ever-increasing longer-horizon work , and how we train Claude itself to be honest, harmless, and helpful. We also put a lot of effort into making sure it's aligned with your intent, even if your intent is expressed in a fuzzy way. Of course, try your best to be specific about what you want, so Claude has all the context — but even when you're not specific, we teach Claude to make good assumptions. It's been a productive partnership. The system prompt has been reduced by 80% — what have you been able to drop? 21:24 So many useful prompting tips in this section! Simon: Thariq, you mentioned this morning that the system prompt for Claude Code has been reduced by 80% because of Claude Fable . Can you go into a little more detail? What kind of things have you been able to drop? Thariq: It wasn't just Fable — it was Opus 4.8 as well, and going forward, future models. We have different system prompts for different models now. One of the patterns we saw is that we were over-constraining Claude. The initial, maybe Opus 4-ish models wanted a lot of examples, and removing examples was extremely helpful , because it was just more creative than the examples we gave it. Simon: That's really interesting, because one of the top prompting tips I give people is: give it examples. If that's no longer true, that kind of breaks my prompting model a little bit. Thariq: Same here — I was surprised to hear that. I think now it's more about the shape of what you give it — the tools you give to Claude, your system prompt, things like that. The other thing we did is try to give it more context and fewer "do not do this" instructions, because that's a very strong impulse for Claude, and especially if it conflicts with user instructions later on, that can be extremely confusing to Claude — "I've got this skill that says this and the system prompt says this." So we try to have fewer hard constraints, more context, and fewer instructions overall . It's definitely a science — it took a bunch of evals to build. Cat: In general, when you're prompting these models, you should always think: are there edge cases to the instruction that I'm giving it? When we went back and reviewed all the instructions in the Claude Code system prompt, we found a few cases where yes, this statement is 90% true, but there's a real 10% of cases where it's not true . We didn't want to constrain the model, or confuse it into thinking it should always do this. One good example is verification. Everyone here wants Claude to verify its work, and we had some instructions in the prompt that said: if you make a front-end change, always verify. But there's a limit to it. If it's changing copy from one string to another string, and the user says "just make a quick fix and update the test," maybe you don't want to verify. So we've adjusted our wording from "always verify, verify, verify" to something like: most of the time when you're doing front-end work you can't fully understand the experience by hitting the backend endpoints, so when you make larger changes to the user experience, please run the app locally. And in fact, that instruction probably isn't even good either, because what is a large change? Maybe it should test small changes too. In general, whenever you give a prompt to the model, you should think about the ways in which it could be misinterpreted by a well-intentioned human , in order to better understand how the model might interpret it — and soften the prompt so that it's actually 100% accurate, because you're giving this prompt to the model 100% of the time. Simon: What's fascinating about that is you're relying on the model's judgment — and that's got to be an Opus/Fable-level thing. Models a year ago did not have the level of judgment necessary to decide whether they were going to test a change or not. But that does break down if you're building for a wide range of models and trying to run the cheaper models for cheaper tasks. Cat: We actually have a different system prompt per model now , for this very reason. It's only our most frontier models that have this 80% token decrease — the older models still have the full system prompt. Simon: Do you think Fable and Opus are smart enough to prompt Haiku with more details, because they understand that Haiku has less judgment, less taste? Cat: We haven't been able to eval it — we don't have any hard data to show it. Thariq: There's a tough thing with smaller models sometimes, because sometimes the larger models can be more token-efficient on a hard problem than the smaller models . So there's a bit of intuition to build there — sometimes you really just want frontier intelligence almost all the time. The Pareto curve shifts, and it's hard to find. Simon: A year ago I did not trust a model to write a prompt. Today the good models are very good at prompting — a lot of my prompts are written by models, which feels absurd but works really well. What helped me come to terms with that was thinking about subagents, which are entirely about a Claude model setting up a prompt for another Claude model. Thariq: Workflows are actually a really good example of this, because it's Claude not just prompting a single subagent, but prompting the orchestration of many subagents, and each one of them gets a very detailed prompt. It's almost a level above just spawning a subagent. I've also been using it on my personal machine, giving it the Gemini API and saying: here, generate images . It's way less lazy than I am at prompting an image model. It's just Claude prompting Claude all the way down. Cat: I think Claude also wrote the prompt for the workflow tool . Simon: I've read that prompt — it's a good prompt. That's actually a frustration I have with Anthropic generally: you publish the prompts for Claude Chat , but you don't include the tool prompts and the Claude Code prompts. I still have to run a proxy to intercept them. I would love it if the Claude Code prompts were deliberately published — they're the documentation. They're how you know what the tool can do and how it works. Cat: I'll write down that feature request. I'll have Claude Tag do it. Interesting to note that OpenAI's prompting best practices for GPT-5.6 includes similar advice for their latest models: Favor leaner prompts Removing repeated instructions and examples and simplifying tool descriptions can improve task performance and token efficiency. In a sample of internal coding-agent eval runs, configurations with leaner system prompts improved evaluation scores by roughly 10–15% while reducing total tokens by 41–66% and cost by 33–67%. What's your bar for introducing a new tool? 28:06 Simon: Claude Code is basically a big bag of tools. What's your bar for introducing a new tool? How do you decide when it's worth doing that additional engineering at that level? Cat: Do you want to take it? You introduced one of the best tools we have. Thariq: My career peaked when I introduced the ask user question tool. It's really hard. Especially for some tools — ask user question is Claude's tool to ask you — so it's hard to eval, and sometimes it's more of a user preference thing. Back then we had fewer evals, so it was very dogfooding based — or "ant fooding," our ant version of that. But overall we've been trying to trend towards fewer tools . The last set of tools we introduced was the task tool, I think — and we try to give Claude more general versions to do things. What's the latest evolution of your file editing tool? 29:03 I have a long-running fascination with file editing tools - they were the subject of the old Aider code editing leaderboard , and I've watched with interest as they've evolved in different coding agents from search-and-replace based to line-number-based to more complicated patterns. The Claude API docs describe a text editing tool that's recommended for building against the API, but Claude Code seems to use slightly different approaches here. Simon: One of the most interesting tools is the file editing tool — you can have file editing as a tool, or you can tell it to use sed and grep and do things that way. What's the latest evolution of your file editing tool? Thariq: We still have one, but for example we removed our grep and other search tools — glob tools — in favor of native bash. Like I said in my talk earlier, the models are kind of more of a biology than a physics , and tool design especially is quite hard. I'm not sure if Cat disagrees and thinks there's a science to the eval of it, but I think tool design is more of an art, maybe — or a biology. Cat: I largely agree, but in general as we introduce more tools, we try to keep the cardinality pretty low and make sure that every tool we add has a distinct function from every other tool, so that Claude can very easily distinguish when to call each . For file edit, the reason we have it is actually because we can render it. We show people when Claude makes a file change, and there's this nice dedicated UI that says: do you approve this edit to this file? The reason we had a dedicated file edit tool was so that we could deterministically know that Claude was making a file change, so we could show people this nice UI. A lot of new users onboarding still really like this experience, so we've kept it around. But for a lot of us who are on auto mode right now — hopefully you're not on YOLO mode — I don't think it actually matters, and we could probably just remove file edit and be totally fine. What's the advice within Anthropic for safely running Claude Code? 30:58 It's the prompt injection question! Who better than Anthropic employees to explain how Anthropic sees the risk of prompt injection attacks causing their Claude Code instances to run amok? It turns out they really trust their auto mode - and see that as the feature that enabled Claude Tag. Simon: Let's talk about safety and security. I am deeply aware of the risks of prompt injection, and there are so many bad things that can happen if somebody else tells my Claude Code what to do. I still mostly run Claude Code in YOLO mode and feel incredibly guilty about it. What's the advice within Anthropic for safely running Claude Code? Cat: Why not auto mode? Simon: I am starting to use auto mode, but I don't understand it enough to get how safe it is. As of maybe three weeks ago, I'm defaulting to auto mode. Cat: Broadly within Anthropic, almost every single person uses auto mode. It is the best way to do long-running work in Claude Code while being safe. We've done extensive bashing. We have thousands of evals. We've commissioned many red teamers to create adversarial environments in order to trick Claude Code into doing bad actions, and we've mitigated every single issue that they found. We're going to publish some evals in the coming weeks, but we've pretty much mitigated every attack. Simon: That is a big claim. Cat: We'll share the evals for it so folks can assess, but we've been extremely diligent about identifying all the ways in which Claude might mess up and then updating auto mode to counter it. It doesn't catch 100% of things — that would be way too strong a claim. But for the main categories of risks that we're concerned about, like prompt injection and data exfiltration, the risks are far lower than the average human reviewer . I am very much looking forward to learning more about their evals and approach to verifying auto mode. Thariq: A little on how auto mode works — it's useful to build this mental model. Whenever Claude is doing a turn, or a bash call, there's a Sonnet classifier that is judging the tool call and also the context of the conversation — your instruction. There are some things around permissions that are dependent on your request: you don't want to give git push permissions all the time, but if you say "push this to GitHub," you want it to do it — and if you say "don't push," you want it to deny it. Auto mode will do that. That particular thing happens to me a lot, where Claude tried to do something because it's very helpful and proactive, and auto mode saw "don't do this" and surfaced it. So it's good at the dynamic permissions that you yourself give inside the prompt, which I think is really important. It also works well with our sandboxing infrastructure , because sandboxing is one of those things where there are so many different edge cases that it's hard for us to deterministically follow them. We have a sandbox, and when something needs to escape the sandbox — like a network request — auto mode can look at that request and ask: does this make sense? — and allow it. Simon: I hadn't realized auto mode is interacting with the networking sandbox as well. Cat: It interacts with any permission prompt the user would otherwise see. Simon: How old is auto mode? As a feature I had access to, it's only a couple of months old, right? (It was first made available to the public on March 24th .) Cat: We've been using it within Anthropic since January , so we've been hardening it for quite a while. Anthropic is extremely focused on safety and security, and we've been working broadly across our alignment and safeguards teams to enable the rollout internally, build out these evals, and make auto mode even more robust before sharing it with the world. Thariq: This is also the reason Claude Tag is so good — Claude Tag uses auto mode . I've heard a lot of build-versus-buy questions about a Slackbot, and I'm like: please, you probably shouldn't build your own AI Slackbot. There are so many attack vectors. You have a feedback channel that users can post feedback into, and now your bot is reading it. The work we've put in with auto mode — and we have a general Swiss cheese defense for security; we also RL against this stuff — I think this is really what makes Claude Tag work . It works seamlessly with your permissions, and you don't want to be prompt injected in your Slack. Are there more security things in the pipeline beyond auto mode? 35:54 Simon: Are there any more security things in the pipeline that go beyond auto mode? Thariq: I think we're very secure. With Claude Tag you can provision your own credentials for Claude , so it doesn't need to act on your behalf — you can have Claude as an identity, and that also makes it easier to audit and inspect what Claude is doing. Simon: Because Claude Tag is influenced by anyone who can talk to it — it's got a much wider pool of people telling it what to do. Thariq: That's right. And of course we have probes as well with Fable, which is a downstream effect of our safety and research work. I think this is the moment where you see Anthropic being an AI safety company really paying off: we really want Claude to be able to run in an aligned way over long periods of time , and auto mode has to be basically flawless for this to work — it's all downstream of our being an AI safety company. Cat: We also launched trusted devices for the remote control users out there who want to be safer. And for all of our remote environments, we support credential injection . If you want Claude Code to be able to access Datadog, but you don't want Claude Code itself to hold the Datadog credential, you can set up our identity and credential management system so that the Datadog credentials are only usable by the agent but not accessible by the agent — we insert them on the fly when the agent tries to make a Datadog request. I really like that credential injection pattern, where Claude Code can access an API via a proxy and that proxy both audits the request and injects the relevant API key - so Claude can access authenticated endpoints without having access to the API credentials itself. How has the past year and a half changed how you think about your own craft? 37:53 Thariq talked about a sense of grief brought on by Fable-class models in his keynote in the morning, and we dived further into that as part of our conversation. I've been calling this Deep Blue . Simon: Let's talk a little bit about the human element. A lot of people are feeling a sense of loss now that so much of what they considered to be their role in building software is being subsumed by the models. How do you think about that? How has the past year and a half changed the way you think about your own craft and the value that you add? Thariq: Cat and Boris are such good reminders that you have to be more ambitious. They're always like: we're growing so fast, we have to be on the edge, we have to do the best work we can. That's a constant reminder for me — any time I'm slow on something, I'm like, okay, can I do it faster? Can I be more ambitious here? And oftentimes the answer is Claude, because Claude is getting better as you go — the last time I tried this, it was with the previous model. On your point about loss: I think this is real. If you're only trying to do the same work you were doing before LLMs, and now it's a prompt, it is, I think, kind of a sad feeling. And the way you offset that is by being more ambitious. I think Jared is such a good example — he hand-wrote all of the Zig code in his Oakland apartment in about a year, barely left his house, and had so much fun doing that. Now I see him rewrite all of Bun into Rust and he's having so much fun doing that — it's so much more ambitious, and that's how he offsets it. Generally it's asking how do I do the bigger thing and do more — I think success is fun . It's changing your ambition. "The way you offset that is by being more ambitious" neatly captures where I've landed on this issue myself as well. Simon: And Cat, what does that look like from a product management perspective? Cat: I feel like the product role just changes every single month. All the PMs on our team are this mix of engineer, designer, PM — most of them actually used to be full-time engineers. For us it really means plugging in whenever there's any kind of gap . If we have an idea and we didn't inspire any engineer to go build it, then we should just build it, put it into a notebook, and inspire people to take it to production. If the designs look a little off, let's take a page that's similar, do a first-pass design, and tag in someone who's very detail-oriented to fill in the gaps . Or if we notice that our team and product adoption is bigger within the company, and more people need to know what's coming down the pipe for Claude Code, Claude Tag, and Cowork — let's automate figuring out our whole launch calendar, let's automate getting those status updates asynchronously so we're not bugging people, and make sure our updates in our internal announce channels are fully detailed and to the point. For us it's very much understanding what the gap is right now between a great idea and getting something to our customers , and how do we automate it as much as possible . This reflects something I've noticed: when you can produce code so much faster, time spent blocked awaiting a decision from someone else becomes a much more notable bottleneck. Engineers who can make product decisions can move a whole lot faster, and the cost of getting one of those decisions wrong is much less prohibitive. What's a moment when Claude has surprised you? 41:50 Simon: What's a moment when Claude has surprised you? When the model did something you didn't think it would be able to do? Thariq: I've posted a lot about Claude video editing, but most recently I gave a talk at the ACM Agentic conference, and I asked, "Hey guys, do you have the edited video? I'd love to post it and share it with my comms team." They said, "Oh, it's taking so long." So I asked for the raw files. They sent me the video of me talking on stage, the video of the deck, and the audio file, and said, "Good luck." I gave this to Claude, along with my HTML deck, and said, " Hey, can you just edit this together? " And what it does is honestly incredible — I'm ready to ship it. It transcribes the entire video. It notices that sometimes the video of my deck is a little weird — there's a popup of an auto-update in the middle — and it goes, " Oh, I probably shouldn't use the video of your deck. What I'm going to do is slice it up, figure out which slide you're on, and use the HTML source instead. " So it displays the HTML source. Then it's got video of me, but I'm only taking up a small part of the stage, so it's cropping dynamically to where I am on the stage — and I'm pacing, so it's tracking me as I pace. And it's transcribing what I'm saying. Simon: This was Fable, right? Thariq: This was Fable, yeah. It was a good prompt, but it was a one-shot prompt. Then I asked it to add some interesting animations and graphics, and I was just blown away. It does ffmpeg, it does Remotion. Here's Thariq's video on how he used Fable to edit Fable's own launch video , and here's that launch video . What can't it do yet? 43:36 I'm embarrased to admit that I've been finding it quite hard to come up with tasks that frontier models like Fable 5 and GPT-5.6 are unable to accomplish. Cat still doesn't rate its UX design skills: Simon: What can't it do? What are the things where you're still disappointed — where you're waiting for Claude Fable 6 to figure it out for you? Cat: I want it to have better design and UX taste. It's now at the point where if I write out a prompt with a detailed spec of how I want a feature to behave, it will usually behave that way. But the paddings might be off, or the interface just isn't delightful yet. It leans on existing best practices for how apps are designed, but for frontier AI products, there are so many new interaction experiences that we have yet to design . Simon: There's an Opus aesthetic — you can look at something and go, "Yeah, that was designed by Opus." It'd be good if we could move beyond that. Cat: Yeah. I'm very excited for future models to hopefully be interaction design thought partners . Thariq: What can't it do? I would love to see it interact more with the real world. Can it solve science? Can it orchestrate the experiments? There's some amount of coding that goes into that, but there's also this other taste of the broader world that it needs. Which parts of Anthropic's culture should other companies steal? 45:11 I figured this would make a great closing question: Simon: Which parts of Anthropic's company culture do you think uniquely help Anthropic be productive with these tools, that other companies should steal? What are the cultural hacks people should be adopting from you? Cat: I'll share one for Claude Tag. Claude Tag works best when you have it in a public channel, and when most of your channels are public. Claude Tag is able to search across all public channels to get as much context as possible to give you the highest-accuracy answer — and it's only able to do this if it has access to everything . Thariq: I mentioned this in my keynote, but it's so important to me I want to re-emphasize it. The co-founders say we don't negotiate against ourselves , and I think this is really important. You can imagine trade-offs in your head and talk yourself out of doing something ambitious — or you can just try to do the ambitious thing. We're so often asking: what if we just did it? Is this a real trade-off or not? And if so, why — where's the proof that it's a real trade-off, and not just something that sounds reasonable? Make the trade-offs show themselves to you. Be as ambitious as you can. What's your favorite absurd thing you've built with Claude, just because you could? 46:46 I couldn't resist throwing in this one as well. Simon: What's one of your favorite absurd things that you've built with Claude, just because you could build it? Thariq: I'm working on a 2D Street Fighter fighting game with me as a character — and my friends as well. It uses Claude Code to prompt Gemini — and honestly the Seedance model is pretty good — to make video animations. It works great; it's so good at prompting, and it can verify the frames to check whether an animation was good. Simon: Is this Street Fighter 2-level 2D sprites you're generating? Thariq: Yeah, exactly — 2D sprites. The animation looks amazing. And it can also figure out hitboxes — it can be like, "Oh, your fist is here, I'll draw the JSON hitbox." It's incredible. Cat: Mine is much more simple. I'm a big rock climber and a lot of my friends climb, so we have this little app we built with Claude Code where we log all the projects we're working on. We also go outdoors together a lot, so we have Claude do all this research with workflows. Workflows is amazing — we brand it as a coding tool, but it's amazing for doing deep research for travel. I also plan our team offsites, and it's good at finding venues that can fit all of us. I use workflows to research all the climbing destinations we might want to go to, and what has direct flights from where all of us are located. It goes to Mountain Project and finds all the climbs at our grade level. It finds the Airbnb. And I don't like hiking, so I care a lot about it having a very short approach — very short walking distance from where the car parks to where the rock actually is — and it filters for this. With existing apps I have to manually click through Mountain Project, but with this I just put in all of our preferences and it's a custom app for us. Simon: So you're basically vibe coding Jira for mountain climbing. Cat: Exactly. Audience: Any plans for eval-building tools and agent observability? 49:23 We had a few minutes at the end for questions from the audience. Audience: Do you have any near-term plans to build more eval tools for us to build eval datasets, and more observability tools to monitor the performance of agents and workflows? Cat: We've considered building eval tools, but I think the limiting factor actually tends to be that it takes a long time for customers to build really high-quality evals . So I think the tooling is less of the constraint, and more the skill set of how you build a great eval. That's an area where we're excited to both invest internally and hopefully share some best practices externally. Audience: How is memory designed today — and would you move from files to a data store? 50:08 Audience (Sai): I'm interested in the memory and the multiplayer. How is memory being designed today? I assume it's around files. And second, have you thought about an orthogonal direction where you would actually need a data store for these memories, instead of files, to scale it better? Thariq: Right now for Claude Tag the memory is channel-specific. Every Claude in that channel has a shared memory, and the instances have a session — but the session can contribute back to main memory. We do a lot of memory research, and it can be kind of unintuitive what the right way to do memory is. We're always running memory experiments. How it works right now in Claude Tag is a markdown file per channel. Tags: ai , prompt-engineering , generative-ai , llms , anthropic , annotated-talks , coding-agents , claude-code , thariq-shihipar , cat-wu

ReadSource

Simon Willison LLMs / 10:29 PM

simonw/pedalican

simonw/pedalican Clearly I wasn't paying attention when these were first announced back in May, but today I accidentally activated a "pet" in Codex Desktop - a little animated robot, reminiscent of Clippy - and then learned you can create your own. So I did, and now I have a cute little pelican on a bicycle bouncing around my desktop giving me updates on my Codex tasks. Your browser does not support HTML5 video. The most interesting thing about this process was watching how the custom pet was created. I told it I wanted a custom pet that was a pelican riding a bicycle and GPT-5.6 Sol xhigh did the rest of the work, using several rounds with gpt-image-2 to generate the necessary sprite assets. I had it make extensive notes and record all of the intermediary steps . My GitHub repo includes every generated image and combined sprite sheet, plus GIFs for each of the animation loops such as this one, called waving.gif : That GIF was compiled from a single image generated by gpt-image-2 that looked like this: And that image was created by executing this prompt against the initial generated character reference image , which was created with this prompt , which has this structure: Create one clean full-body reference sprite for Codex pet Pedalican. Pet identity: A compact adorable baby pelican with a round cream-white body, soft coral-orange bill and feet, riding a tiny sky-blue bicycle [...] Place a single centered pose on a perfectly flat pure magenta #FF00FF chroma-key background. Keep the full pet visible, compact, readable at 192x208, and easy to animate. [...] I've been looking out for ways to use image generation to create simple game-ready sprites, so I spent some time digging into this mechanism to see how it works. The key implementation details are open source - these two skills in particular, both Apache 2.0 licensed: hatch-pet from openai/skills imagegen from openai/codex And yes, GPT-5.6 Sol did come up with the name "Pedalican". I like it! Tags: ai , prompt-engineering , generative-ai , llms , text-to-image , pelican-riding-a-bicycle , codex

ReadSource

Product Hunt AI / 9:35 PM

html.contact

A full form backend you can test before paying Discussion | Link

ReadSource

Simon Willison LLMs / 5:19 PM

datasette-agent 0.3a0

Release: datasette-agent 0.3a0 New tool, execute_write_sql , which requests user approval and then writes to a database - taking user permissions into account. #27 I added a mechanism for asking user approval in datasette agent 0.2a0 . The new execute_write_sql tool can now prompt the user for all kinds of useful operations. Here's an example where I add some pelican sightings to my pelican_sightings table: The new version also enhances the datasette agent chat terminal mode to support approvals, and adds several new options including --unsafe mode for auto-approving them: datasette agent chat can execute tools that require user approval. #30 Three new options for datasette agent chat - --root to run as root, --yes to approve all ask user questions, and --unsafe for both. Tools can now provide plain text alternatives to HTML, for display in the datasette agent chat CLI. #31 The datasette agent chat content.db -m gpt-5.5 --unsafe command can now be used to chat directly with a specific database and directly modify it through prompts like "create a notes table", "add a note about X" etc. Tags: projects , ai , datasette , annotated-release-notes , generative-ai , llms , llm-tool-use , datasette-agent

ReadSource

Simon Willison LLMs / 11:57 PM

Claude Fable 5.1 made me a really nice animated pelican

Today is Claude Fable (and Mythos) 5.1 day . Anthropic say that Fable 5.1 "sets a new standard for coding, knowledge work, and long-running problem-solving tasks". Their announcement spends a notable amount of time on scientific research, boasting of a 52.6% score on the brand new Terminal-Bench-Science 0.1 benchmark (first announced on August 27th ), up from 24.7% for Fable 5, 29.0% for Opus 5 and 22.4% for GPT-5.6 Sol. Other benchmarks show slightly improved scores, but none as impressive as the Science one. But how well can it pelican? Back in July I wrote about how I was losing faith in the pelican benchmark - its connection to how good the models were at other tasks didn't seem to hold as strongly as it did back in 2025 . The most interesting insights I get from it now are comparisons within model families, and particularly comparisons for the same prompt at different reasoning effort levels. Fable 5.1 has five reasoning levels: low, medium, high, xhigh, max - and no option to turn off reasoning entirely. I fixed an issue in llm-anthropic which caused reasoning traces not to be correctly recorded, then ran some prompts. Here's the full set of pelicans for all of the reasoning levels, each with the full reasoning transcript. I'll replicate them here: Low and medium, both without reasoning? Next, a bit of a mystery. This is what I got for effort low : The transcript doesn't show any summarized reasoning tokens, and the output token count is 1,998. With Claude that output token count includes reasoning tokens. It took 23.8 seconds and cost 10.017 cents . I bumped that up to medium and got this: Weirdly, that one also shows no reasoning text and used 1,977 output tokens - 21 tokens less than low . It took 23 seconds and cost 9.912 cents . So for this particular prompt ("Generate an SVG of a pelican riding a bicycle") Fable 5.1 appeared to skip reasoning entirely at both low and medium settings. High Here's high - 29.6 seconds, 2,612 output tokens, 13.087 cents : This one did do a bit of reasoning, summary here : I'm planning the SVG layout for a pelican riding a bicycle, with a sky and ground background, a bicycle with two spoked wheels, frame, seat and handlebars, and a white-bodied pelican with a long neck and orange beak positioned on top. Really not much difference from low and medium , though. Extra High At xhigh things got radically different. 36,767 output tokens, 7 minutes 51 seconds, $1.83 ! The reasoning trace is pretty lengthy , and includes details like this: Adding the eye, wings stretching down to the handlebar grip, orange legs reaching to the pedals, and a small tail feather, while keeping the pelican intentionally oversized compared to the bike for comic effect. [...] I'll accept the slight thickness as charming rather than overengineering it. Max Setting effort to max gave me the best pelican I've seen from any of Anthropic's models. 65,927 output tokens, 13 minutes and 54 seconds, $3.30 : There's a lot to like about this. The background is tasteful, the legs are clearly on either side of the frame, the feet are on the pedals, the wing is on the handlebars, the pelican has a cute blue hat and there's a basket with a fish. It's still not showing nearly the same level of flair as Gemini 3.7 Flash , but I didn't ask for flair - I asked for an SVG, and that's what I got. Some highlights from that reasoning trace : Adding pedal shapes near both feet, with the far foot on the second leg partially visible behind the frame. I'm considering whether to add a small scarf or cap for extra character, but leaning toward keeping it simple to avoid clutter. Now I'm debating a bicycle helmet on the head versus the pelican's signature crest—the beak and pouch already read clearly as "pelican," so a helmet could reinforce the bicycle theme without losing identity, though it might compete with the crest for visual space. I realize the beak at (484,84) would overlap with the dome helmet, so I need to shrink the helmet so it only covers the top of the head, adjusting its arc endpoints to sit higher and narrower so the beak can attach cleanly at the front without collision. [...] I'm adding a darker tip region to represent the primary feathers, then reconsidering the trailing edge to include scalloped feather curves instead of one smooth line for a more natural look. [...] Now I'm checking the vent line placements on the helmet, making sure they sit far enough inside the helmet's edge given the stroke width and rounded caps, and confirming each vent stays within the helmet's circular boundary. [...] I decide skipping a handlebar bell and tire highlights since they're unnecessary additions. Now I'm reconsidering the front fork's curve — the current control point pulls the shape backward when it should bow forward for a proper rake, so I need to shift the control point rightward to fix the fork's lean. OK, let's animate it On Hacker News, swalsh commented on that Max pelican: Now that it's a solved benchmark, can we get the animated version? I didn't want to spend another $3 so I took the Max pelican and piped it into the default thinking level of High: llm logs -cx | llm -m claude-fable-5.1 -s ' animate this ' 6,121 input, 26,201 output = $1.37 . The result looked like this , exported here as video since some people have trouble viewing animated SVGs: Your browser does not support HTML5 video. The wheels in the video are rotating in the wrong direction, but I think that's an artifact of the conversion to MP4 - they seem to be going in the correct direction in the original SVG. Tags: ai , generative-ai , llms , anthropic , claude , pelican-riding-a-bicycle , llm-reasoning , llm-release

ReadSource

Simon Willison LLMs / 10:00 PM

Qwen 3.8 27B is excellent, but it defaults to wildly overthinking things

Friday's big release was Qwen 3.8 27B , an Apache 2 licensed 27B parameter vision-capable LLM from Alibaba's Qwen research lab. I've been looking forward to this one: 27B is an excellent size for running a model on a reasonably specced laptop, and its predecessor Qwen 3.6 27B was impressive. Qwen's self-reported benchmarks for this model are eye-opening. They show a boost from both Qwen 3.6 27B and the closed-weight Qwen 3.7-Plus, which was one of Qwen's strongest models of any size as recently as May this year . It will be interesting to hear what independent benchmarks have to say about the model. I've been running the model on two different machines: my 128GB M5 Max MacBook Pro, and an NVIDIA DGX Spark . On both machines I'm running LM Studio and their 17GB Q4_K_M quantized build . I also tried using llama-server directly on the Spark. The default of extra high results in spectacular over-thinking Qwen's documentation describes the model as defaulting to xhigh for the reasoning effort, and the LM Studio GGUF I've been trying preserves that default: Qwen3.8 comes with official support for reasoning_effort , which can be used to adjust reasoning depth and control cost: xhigh (default): for complex tasks demanding thorough analysis medium : balancing accuracy and speed low : efficient reasoning optimizing for speed and cost This is a hilarious default. It's absolutely not a good way to run the model, especially on consumer hardware. I've been finding the results extremely entertaining. I quickly ran into problems with LM Studio's default context limit of 8,192 tokens - Qwen was using them all up thinking about even the most mundane of problems. I loaded the model with the full 262,144 maximum context length and that problem went away. Here's the pelican riding a bicycle SVG I got from my first attempt with that increased context length. It took 21 minutes to generate, using 22,276 reasoning tokens to produce 3,223 tokens of output. You can read the reasoning trace here . This is by far the best pelican SVG I've been able to generate with a model that runs on a local machine - and this Qwen is pretty small, just a 17GB file on disk. There's a lot to like about this: The bicycle frame is the right shape It has legs on each side of the bike - that's very rare Good, clear pelican pouch The wings extend to touch the handlebars! The motion lines are behind, not in front It has a tasteful background - nice sun, clouds, hill, flowers and grass. Was that worth waiting 21 minutes for? Absolutely not. Here's that same prompt run with reasoning turned off - transcript here . This one produced 3,715 tokens and took 137s - just over two minutes. And just for completeness, I used OpenRouter to run the same prompt through the much larger Qwen 3.8 2.4T-A95B (released last week ) and got this snazzy animated SVG : Your browser does not support HTML5 video. I said Qwen at xhigh has a tendency to over-think things, but how bad really is it? I tried a much simpler prompt, again with that default extra high setting: draw an svg of a circle Qwen's reasoning trace started like this: The user is asking for an SVG drawing of a circle. Simple request — but I want it to be a carefully crafted piece. Let me make something that goes beyond just <circle> : a single self-contained SVG file with character — maybe a geometric "circle study," with subtle animation, layered rings, and a distinctive palette. Keep the scope right: they asked for an SVG of a circle. So the core is a circle. But I can add craft: concentric guide circles (like a compass/geometry drawing), tick marks, a soft gradient fill on the main circle, restrained ambient motion (a slowly rotating dashed ring, pulsing glow). Respect prefers-reduced-motion? That's more for CSS; SVG SMIL or CSS inside SVG will do. Palette options: deep teal ink on warm paper? Or bold vermilion circle on off-white with navy construction lines — Bauhaus/compass-drawing vibe. Let me go with a "geometric study" look: cool slate background, or bright paper white? Paper white is fine if it's not the cream-and-terracotta combo. [...] Several minutes later it produced this absolutely beautiful animated circle, which was entirely not what I had asked for! Your browser does not support HTML5 video. My strong recommendation: ignore that default. Run Qwen 3.8 27B on low or even no reasoning levels at first. It's a great model, but wow that default setting is a bad place to start. It's very good at bounding boxes A fun way to test a vision model is to see how well it can return bounding boxes around items in a photograph. I've seen previous Qwen models deal well with this, so I decided to put it to the test drawing bounding boxes around some pelicans. I've seen asking for 0-1000 scale produce good results in the past. I tried this: llm -a https://static.inaturalist.org/photos/714731804/large.jpg \ -m lmstudio/qwen/qwen3.8-27b \ ' Return JSON bounding boxes for the pelicans in this photo, 0-1000 scale for each dimension ' Here's the reasoning trace , which produced this: [ { "bbox_2d" : [ 195 , 290 , 370 , 780 ], "label" : " pelicans " }, { "bbox_2d" : [ 445 , 320 , 675 , 850 ], "label" : " pelicans " } ] This is such a good match . Here are those boxes rendered on top of the photo: Building a tool to label bounding boxes That visualization of the bounding boxes was taken using a new custom tool that I had Qwen 3.8 27B build for me, running offline on my laptop. I forgot to dial down the thinking effort so it was massively over-engineered , but it did manage to produce this full interface from this single prompt : [ {"bbox_2d": [195, 290, 370, 780], "label": "pelicans"}, {"bbox_2d": [445, 320, 675, 850], "label": "pelicans"} ] Build an HTML page which has an input box for accepting the URL to an image and a textarea for accepting the above style of JSON. It appends the image to the page, measures its width and height, then treats the coords in the bbox_2d as scaled from 0-1000 and scales them against the actual width and height, then it renders labelled boxes over the image. This screenshot shows one of the features I did not ask for - a demo scene, for if you don't have a photograph to test the tool with: Here's the relevant segment of the thinking trace, where it decided to draw its own pelicans purely because I had used the label "pelicans" in the example JSON I gave it in the prompt: Also a "load sample" that uses a known image? Can't depend on external images, but… the image URL input is user-provided; I could add a "try with sample" button [...] Hmm, I can draw a simple scene on canvas, export it as a data URL, and load it into the image — that's self-contained and demo-able! [...] But the user's coords are for an actual pelican image; a generated placeholder can still demo the scaling. Generate a 1000x1000 placeholder: gradient water + two blob-like "pelican" silhouettes placed at the given bboxes (using the same scale — cute: silhouettes at the exact 0-1000 positions, showing the boxes align). This makes for a fun, self-contained demo. Keep it simple: sky gradient, sun, water, two pelican-ish shapes (ellipse body, circle head, beak). Place at bbox centers. (I'm slightly nervous that models around the world might have a bias towards drawing pelicans at any chance they can get, brought on by nearly two years of exposure to my own stupid benchmark.) Is all that over-thinking necessary? Maybe it is, at least a bit. I tried with reasoning turned off and got this version , ( transcript here ), which nearly works but shows the boxes in the wrong place: So without reasoning it didn't quite one-shot a working tool. I'm sure it could get there with some follow-up prompts, but this is a good example of how reasoning can make a difference. Yes, it can drive coding agents One of the biggest questions around local models is whether or not they have enough horsepower to successfully run a coding agent loop. Coding agents require long context, strong code generation support and reliable tool-calling. On paper Qwen 3.8 27B has all three of these, so is it up to the task? My initial experiments with Pi have been very promising. I chose Pi because it has a shorter system prompt than most other options, making it a better fit for trying out smaller models. I configured Pi to use Qwen 3.8 27B running in LM Studio on the Spark (shared via tailscale serve ) by adding this to ~/.pi/agent/models.json : { "providers" : { "spark" : { "baseUrl" : " https://spark-18b3.tail68a31.ts.net/v1 " , "api" : " openai-responses " , "apiKey" : " dummy " , "models" : [ { "id" : " qwen3.8-27b " , "reasoning" : true } ] } } } Then ran pi --provider spark --model qwen3.8-27b in my ~/dev/datasette folder and prompted: how does auth work? After a sequence of reasoning and tool calls that accessed a bunch of different files it produced this reply , which is very solid. Just one problem: I wanted to share that transcript. So I pointed Pi and Qwen 3.8 27B at the JSONL transcript file in ~/.pi/agent/sessions/--Users-simon-Dropbox-dev-datasette-- and prompted: Write Python code to convert this jsonl to markdown And it built and tested this pi_jsonl_to_md.py , which did exactly what I needed. Here's that session transcript , published using the tool that it created. The quest for speed So far this is all looking very promising. We have a 17GB model that runs on high-end consumer hardware and can write code, drive tools, annotate images and generally do everything that I need from an LLM for getting real work done. There's one very significant catch: it feels slow - especially when it starts over-thinking, but even without that it's not particularly sprightly. I've been getting around 15-30 tokens a second from LM Studio. That's not terrible, but it's slow enough that it's going to be hard to win me away from hosted API models, which can return results a whole lot faster. Artificial Analysis track token speed and show OpenAI 5.6 Sol at 74 tokens/second and 5.6 Luna at an impressive 184/second. The good news is that the community have been exploring ways to speed things up since the model was first released two days ago. One of the most promising optimizations is baked into the model itself. Qwen supports Multi-Token Prediction , an architecture trick where a cheaper mechanism guesses several tokens ahead and the main model can then quickly verify if the guesses were correct. This can have quite a dramatic effect on inference performance. Based on this tweet from llama.cpp creator Georgi Gerganov I tried running the model with MTP like this on the Spark: llama serve \ -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M \ -hfd ggml-org/Qwen3.8-27B-GGUF:Q4_0 \ --spec-default \ --spec-type draft-mtp \ --reasoning-preserve And sure enough, this gave me a significant boost. I had GPT-5.6 in Codex run a comparative benchmark on the Spark and the --spec-type draft-mtp server outperformed the LM Studio default GGUF by around 72%. I expect we'll see a whole lot more innovation around serving this model faster over the next few weeks. The MLX community likely have some tricks brewing as well. Some observations The fact that a 17GB file can do all of this stuff on my home machines is a miracle . Once again, I'm delighted and amazed at how much progress local models have made this year. A year ago this would have been competitive with the best and most expensive of the proprietary models - today it can run on a capable laptop. The only thing holding this back from being a daily driver is performance. It feels pretty slow on both the M5 Mac and the DGX Spark. That's the catch with these dense (non-Mixture-of-Experts) models - they require a whole lot of memory bandwidth to perform well, and neither of the machines I have access to are top performers in that regard. The most important thing about Qwen 3.8 27B is what it demonstrates . We can have an open weights general purpose model with a long context, effective tool calling, strong vision ability, and competent code generation, and we can fit the whole thing in just a 17GB file. The models at this size continue to get better at an impressive rate. We don't need to spend half a million dollars on datacenter-class hardware just to run a competent model. Tags: ai , generative-ai , local-llms , llms , qwen , pelican-riding-a-bicycle , llm-reasoning , llama-cpp , llm-release , coding-agents , lm-studio , ai-in-china , nvidia-spark , pi

ReadSource

Simon Willison LLMs / 7:18 PM

Moonlight & Mayhem (Raccoon Heist by Codex + GPT-5.6 Sol Ultra)

Moonlight & Mayhem (Raccoon Heist by Codex + GPT-5.6 Sol Ultra) On Wednesday I wrote about One-shotting a Raccoon Heist game using Claude Fable 5 , where I had Claude Fable 5 build a full working game from a premise I generated with GPT-3 and DALL-E four years ago . I decided to pose the exact same prompt to Codex Desktop running GPT-5.6 Sol Ultra - the mode where Sol makes aggressive use of sub-agents - to see how it would do. It produced a much better game! Here's Moonlight & Mayhem - GitHub repository here , including the textures and prompts it generated using gpt-image-2 . Your browser does not support HTML5 video. The original GPT-3 generated game description included: In “Raccoon Heist”, you and your team of thieving raccoons are tasked with pulling off a series of daring heists. From robbing banks to stealing priceless art, no job is too big or too small for your furry crew. Fable's version had you as a single raccoon running around a back yard collecting coins and fish. GPT-5.6 Sol has you in a museum, rescuing your two other raccoon crewmates in order to stack on top of each other and bust the golden sardine out of its case. Much more heisty! There was one catch though: the version produced from the one-shot prompt had a bug where each raccoon had an eyeball that was enlarged to the size of a giant sphere floating over their head! You can play that version here . Despite reviewing screenshots during development Codex failed to spot and correct this bug. I fixed it by prompting: Why do the raccoons have huge black spheres on them? And then: Fix it Which resulted in this fix . I shared the full Codex transcript in the repository - I wish Claude Code had the same "copy as Markdown" feature. Codex spent 52 minutes on the project. Here's the AgentsView cost estimate for that session if I had been paying full API prices as opposed to using my monthly Codex subscription: Tags: game-design , ai , openai , generative-ai , llms , coding-agents , codex , gpt

ReadSource

Simon Willison LLMs / 9:15 PM

smevals - a small eval suite for evaluating models, prompts, and harnesses

smevals - a small eval suite for evaluating models, prompts, and harnesses I've been working with Jesse Vincent's Prime Radiant applied AI research lab building out this evals framework to help answer questions about the capabilities of different models. The result is smevals , a new tool for running small eval suites across different model configurations and grading the results. The blog entry describes the tool in detail. Here's the 10 second version: Tell your coding agent to run uvx smevals docs to learn the tool (this outputs the README ) Then tell it to build you an eval suite Once you've created an eval - which takes the form of a directory with some YAML files - you can run it against models like this: uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6 Runs are treated separately from grading operations - you can grade your runs (against your defined set of checks) using: uvx smevals grade path-to-eval/ Then you can run a localhost web server to explore the results: uvx smevals serve path-to-eval/ Or run the smevals build command to build that report as static HTML, which you can then host anywhere. Here's an example showing an eval suite I built to evaluate how well models can write haikus. The most time-consuming part of this project was figuring out the vocabulary for it! Here's what I settled on, quoted from the announcement: An eval is a collection of challenges designed to answer a question about a model, for example, how good is that model at generating SVGs? Each eval is a collection of tasks . A task is a specific challenge, for example "Generate an SVG of a pelican riding a bicycle". When you run the eval you do so against one or more configs . Each config specifies a model to be evaluated, but may also include other parameters to test, such as different system prompts, model parameters, or agent harnesses. A run records what happened when a specific config was used to execute a specific task. A runner is the script that executes a run. Once you have collected one or more runs, you need to evaluate the results to see how well the model (or config) did. This is done by a grader , which produces a grade . Each grader runs a sequence of checks . These can be simple operations, like checking for a specific string in the output, or confirming that the output is valid XML. They can also be more complicated custom operations (implemented as scripts called checkers ), including using other models to answer questions about the run. I've been trying to figure out an approach I like for evals for several years now. smevals is my third iteration on the idea and it feels right to me. I'm looking forward to expanding this more in the future, as well as pointing it at some of my own projects. Tags: projects , ai , generative-ai , llms , llm , evals , jesse-vincent

ReadSource

Hacker News AI / 7:40 PM

Hacker News discussion: Write video in HTML, render MP4 in the browser

Hacker News readers are discussing "Write video in HTML, render MP4 in the browser" with 3 points and 1 comments.

ReadSource

Simon Willison LLMs / 10:34 PM

DOOMQL

DOOMQL Peter Gostev built this using GPT-5.6 Sol. This is a lot of fun: DOOMQL started with a deliberately unreasonable question: what if SQLite were the game engine, not merely the place where a game stores data? The result is a small, original Doom-like game in which SQL owns movement, collision, enemies, combat, progression and every RGB pixel on screen. It's implemented as a Python terminal script - I tried it out like this: cd /tmp git clone https://github.com/petergpt/doomql cd doomql uv run host/doomql.py Here's the huge SQL query that implements a full ray tracer in SQLite using a recursive CTE. Running the above script creates a /tmp/doomql/.doomql/doomql.sqlite SQLite database, which you can explore using Datasette like this: uvx --prerelease=allow --with datasette-apps datasette \ /tmp/doomql/.doomql/doomql.sqlite \ -p 4444 --root --secret 1 --internal internal.db The --with datasette-apps option installs the new Datasette Apps plugin, which supports creating custom HTML+JavaScript apps that can run SQL queries directly within the Datasette interface. I created a new app, pasted the copy-paste prompt into Claude chat (Fable 5) and told it : Build an app that displays the current state of the screen using the frame_pixels view with its x, y, r, g, b columns. have it refresh once a second. This got me a working HTML+JavaScript app inside Datasette that could reflect the current state while I played the game in my terminal. Then I added: add a minimap And now my Datasette App looks like this: Here's the HTML app code - paste that into your own Datasette instance (using the uvx --with datasette-apps recipe from above) to try it yourself. Via @petergostev Tags: games , sql , sqlite , ai , datasette , generative-ai , llms , ai-assisted-programming , gpt , datasette-apps

ReadSource

Simon Willison LLMs / 11:58 PM

Datasette Apps: Host custom HTML applications inside Datasette

Today we launched a new plugin for Datasette, datasette-apps , with this launch announcement post on the Datasette project blog. That post has the what , but I'm going to expand on that a little bit here to provide the why . The TL;DR Datasette Apps are self-contained HTML+JavaScript applications that run in a tightly constrained <iframe> sandbox hosted on your Datasette application. They can use JavaScript to run read-only SQL queries against data in Datasette, and can run write queries too if you configure them with some stored queries . Here's a very simple example and a more complex custom timeline example - the latter looks like this: Apps are allowed to run JavaScript and render HTML and CSS. They are limited in terms of access - the <iframe sandbox="allow-scripts allow-forms"> they run in prevents them from accessing cookies or localStorage and they also have an injected CSP header (thanks to this research ) which prevents them from making HTTP requests to outside hosts, preventing a malicious or buggy app from exfiltrating private data. Datasette Apps started out as my attempt at building a Claude Artifacts mechanism for Datasette Agent , but I quickly realised that the sandboxed pattern is interesting for way more than just adding custom apps in a chat interface and promoted it to its own top-level concept within the Datasette ecosystem. They're also a fun way to turn my multi-year experiment in vibe-coded HTML tools into a core feature of my main project! You can try out Datasette Apps by signing in with GitHub to the agent.datasette.io demo instance. Why build this? Since the very first release, Datasette has offered a flexible backend for creating custom HTML apps via its JSON API. One of my earliest Datasette projects was an internal search engine for documentation when I worked at Eventbrite - it worked by importing documents from different systems into SQLite on a cron and then serving them through a Datasette instance with a custom HTML+JavaScript search interface that directly queried the Datasette API. I had client-side JavaScript constructing SQL queries, which originally was intended as an engineering joke but turned out to be a really productive way of iterating on the app! That project, combined with my experience building my HTML tools collection and my experiments with Claude Artifacts , has convinced me that adding a Datasette-style backend to a self-contained HTML frontend is an astonishingly powerful combination. Imagine how much more useful Claude Artifacts could be if they had access to a persistent relational database. That's what I'm building with Datasette Apps! Neat ideas in Datasette Apps Here are a few of the ideas and patterns I've figured out building this which I think have staying power. <iframe sandbox="allow-scripts" srcdoc="..."> + <meta http-equiv="Content-Security-Policy" content="default-src 'none'; script-src 'unsafe-inline'; style-src 'unsafe-inline'; img-src data: blob:;"> This is the magic combination that makes Datasette Apps feasible in the first place. I need to run untrusted HTML and JavaScript on a highly sensitive domain - an authenticated Datasette instance can contain all sorts of private data. The sandbox= attribute lets me run that untrusted code in a way that cannot interact with the parent application - it can't read the DOM, or access cookies, or steal secrets from localStorage . It can however use fetch() and friends to load content (or exfiltrate data) from other domains. But... it turns out if you start an HTML page with a <meta http-equiv="Content-Security-Policy"> header you can set additional policies that lock down access to other domains. I was worried that malicious JavaScript would be able to update or remove that header but it turns out that doesn't work - once set, the CSP policy is immutable for the content of that frame. Locked down APIs with postMessage() and MessageChannel() Having locked down those iframes to the point that they couldn't do anything interesting at all, the challenge was to open them back again such that they could run an allow-list of operations, starting with read-only SQL queries against specified databases. I built the first version of this with postMessage() , which allows a child iframe to send messages to the parent window. I created a simple protocol for requesting that the parent run a SQL query - the parent could then verify it was against an allow-listed database before executing it. One of the LLM tools, I think it was GPT-5.5, suggested that postMessage() on its own can be exploited if the iframe somehow loads additional code from an untrusted domain. I don't think that applies to Datasette Apps, but I also believe in defense in depth, so I had GPT-5.5 help me port to a MessageChannel() based transport instead. MessageChannel() has the advantage that if a page navigates to somewhere else the channel closes automatically, removing any chance of executing commands sent from an untrusted external page. Visible logs, for queries and errors If you navigate to the timeline demo and search for the string usercontent you'll pull in some search results that embed images from the user-images.githubusercontent.com domain. This domain is not in the CSP allow-list, so it trips an error. Those errors are captured and transmitted back to the parent frame, where they can be displayed in a useful error log. This is meant to make hacking on apps more productive by surfacing otherwise-invisible problems. I built an experiment demonstrating that you can even turn this into a one-click-to-allow mechanism for building the CSP allow-list based on what breaks, but I haven't integrated that idea into datasette-apps just yet. SQL queries are also visibly logged - scroll to the bottom of the timeline page to see that in action. Stored queries for write operations I want apps to be able to conditionally write to the database, but this is an even more dangerous proposition than SQL reads! My solution involves Datasette's stored queries feature, rebranded from "canned queries" and given a major upgrade in the recent Datasette 1.0a31 - work that was directly inspired by Datasette Apps. Users can create a stored write query that performs an insert or update, then allow-list that specific query for an app to use. Usage from code inside an app looks like this: const result = await datasette . storedQuery ( "todos" , "add_todo" , { title : "Buy milk" , due_date : "2026-06-20" , priority : "high" , completed : false } ) ; I'm only just beginning to explore the possibilities this unlocks myself, but my goal is to support full read-write applications built safely as Datasette Apps. Copy and paste a prompt to build an app The Datasette Apps plugin has no dependency on LLMs at all, but these self-contained apps are the perfect shape to be written by a modern LLM. The create app form includes a copyable prompt at the end. This prompt has everything a model needs to know to build a new app, including the schema of any selected databases. This means you can click "copy", paste it into ChatGPT or Claude or Gemini, tell it what you need, and there's a good chance the model will spit out the code necessary to build the app. If you have Datasette Agent installed your AI assistant will also gain tools to both create new apps and edit existing ones, Claude Artifacts style. Built with so much AI assistance Datasette Apps started life back in April as datasette-agent-artifacts , a plugin I have since renamed to datasette-agent-edit keeping only its editing tools . I built that as one of the first plugins for Datasette Agent , to help get the plugin hooks into the right shape. That first prototype was mainly built using Claude Opus 4.6 in Claude Code. When I switched track to Datasette Apps I started with a plan constructed using Codex Desktop and GPT-5.5 xhigh, based on extensive dialog and feeding in both datasette-agent-artifacts and other prototypes I had built. Most of the work that followed stuck with Codex, but in the few short days that we had access to Claude Fable 5 I had it run a security evaluation of the product (an ability that would get it banned by the US government shortly afterwards) and it found a very real problem. I was allowing users to allow-list CSP hosts for their apps, but Fable pointed out the following attack: A less privileged user with create-app permission creates an app that queries SQLite for all available tables and selects and exfiltrates all of the data to a host they had allow-listed via CSP. They then trick an administrator user with access to private data into visiting their app. ... and the app can now run queries as that user and steal their private data! That's clearly unacceptable. I fixed it by restricting the ability to allow-list any domain to a new apps-set-csp permission, which is intended just for trusted staff. Site administrators can also configure Datasette with a list of allowed_csp_origins , which regular users can then select. This means you can do things like allow cdnjs.cloudflare.com and your users will be able to build apps that load extra JavaScript libraries from the cdnjs CDN. I've reviewed Datasette Apps extremely closely, especially the security-adjacent parts of it. The critical sandbox and CSP configuration are based on multiple AI-assisted prototypes and tests. It's looking good so far I'm really pleased with this initial release. Datasette is growing beyond its origins as an application for serving read-only data into a much richer ecosystem of tools for doing useful things with that data once it has been collected. Datasette's roots are in data journalism. I've always been interested in the question of what comes next after a journalist gets their hands on a giant dump of data about the world. Datasette supports exploring and publishing it. Datasette Agent adds interrogating it with AI assistance. Now Datasette Apps expands that to building custom interfaces and visualizations to help unlock the stories that are hidden within. Tags: iframes , javascript , projects , sandboxing , ai , datasette , generative-ai , llms , ai-assisted-programming , content-security-policy

ReadSource

Simon Willison LLMs / 11:58 PM

GLM-5.2 is probably the most powerful text-only open weights LLM

Chinese AI lab Z.ai released GLM-5.2 to their coding plan subscribers on June 13th, and then yesterday (June 16th) released the full open weights under an MIT license. Similar in size to their previous GLM-5 and GLM-5.1 releases this is a 753B parameter, 1.51TB monster - with 40 active parameters (Mixture of Experts). GLM-5.2 is a text input only model - Z.ai have a separate vision family most recently represented by GLM-5V-Turbo , but that one isn't open weights. GLM-5.2 has a 1 million token context window, up from GLM-5.1's 200,000. The buzz around this model is strong. Artificial Analysis, who run one of the most widely respected independent benchmarks: GLM-5.2 is the new leading open weights model on the Artificial Analysis Intelligence Index . GLM-5.2 is the leading open weights model on the Intelligence Index v4.1. At 51, it leads MiniMax-M3 (44), DeepSeek V4 Pro (max, 44) and Kimi K2.6 (43) They did however find it to be quite token-hungry: GLM-5.2 uses more output tokens per task than other leading open weights models: the model uses 43k output tokens per Intelligence Index task, up from GLM-5.1 (26k) and above MiniMax-M3 (24k), Kimi K2.6 (35k) and DeepSeek V4 Pro (max, 37k) The model is also now ranked 2nd on the Code Arena WebDev leaderboard , behind only Claude Fable 5. That leaderboard measures "front-end web development tasks, including agentic coding workflows". I'm impressed to see it rank so highly given the lack of image input, which I had incorrectly assumed was a key part of building a truly great frontend coding model. I've been trying it out via OpenRouter , which has it from 9 different providers, almost all of which are charging $1.40/million for input and $4.40/million for output. For comparison, GPT-5.5 is $5/$30 and Claude Opus 4.5-4.8 is $5/$25. Excellent pelican, disappointing opossum GLM-5.1 gave me one of my favorite pelicans and my all time favorite opossum (for the prompt "Generate an SVG of a NORTH VIRGINIA OPOSSUM ON AN E-SCOOTER".) Interestingly, in both of those cases the model chose to return SVG wrapped in an HTML document that added additional animations using CSS. Let's try GLM-5.2. For "Generate an SVG of a pelican riding a bicycle" I got this : It's a self-contained fully animated SVG, and the animations aren't broken! Often I'll see eyes falling off or wheels rotating independently of the bicycle but here everything works great. It's a very nice vector illustration of a pelican too. Very impressive. Sadly, the NORTH VIRGINIA OPOSSUM ON AN E-SCOOTER did not come out nearly as well : This is such a step down from GLM-5.1! As a reminder, that possum looked like this: 5.2 didn't even try to animate it. Tags: ai , generative-ai , llms , pelican-riding-a-bicycle , llm-release , openrouter , ai-in-china , glm

ReadSource

Latest story in this edition: 11:56 PM

Back to front page