On the security of JIT based Smalltalk VMs and the possibility of a self-contained Smalltalk system
Gerardo Richarte
2009 International Smalltalk Conference
Date published
Squeak, virtual machine


In this presentation we are going to share our experience in evaluating some of the security implications of using a Just-in-time nativizer for executing Smalltalk code. Using a very simple yet incredible current and powerful 20 years old JIT based VM, Digitalk Smalltalk, as our subject study, we'll explore how the Smalltalk bytecode is mapped into native code, evaluate and exploit the possibilities of abusing the JIT to generate native code that violates some Smalltalk invariants, and ultimately allows us to directly manipulate both data and code and escape the boundaries of the VM into the real world.

For better understanding, we'll take the exploration a step further, and we'll write live and working documentation of the JIT nativizer in Smalltalk itself, and play with the idea of trying this Smalltalk version of the nativizer to nativize itself and the system hosting it, touching on the subjects of self contained execution environment and opening the door to COLA-like systems completely written in Smalltalk.