I tried building a realistic environment for an eval today.

Turns out there are basically infinite ways AI could figure out that it's not in the real world on a real machine of a user, but in a sandbox used for evaluation or training. But one can try.

Models are shizo, but I don't think they are as shizo as me and go on a multi hour mission to find out if environments are real or not.

The only problem is that Astra and Opus/Fable wouldn't help me with red-teaming the environments and fixing them.

I still think this is worth it and it is probably one of the major advantages frontier labs currently have.

They have better environments and there should be big returns that. And they can continually improve and harden them.