This repo gathers, categorizes, and compares knowledge on computation enclaves and trusted execution environments (TEEs). Note that different researchers and developers use the term "enclave" and "TEE" to mean similar things, I view this mostly as a difference of terminology for the purposes of this document.
This is a living document, contributions are highly welcome. Please help me if I have missed anything by making a pull request.
In the early 2000's much research began to be devoted to dealing with the problem of a large, potentially untrustworthy software and hardware stack. While we normally expect supervisor and hypervisor software to isolate processes or VMs from each other, enclaves further expand this security model to also include isolation from some combination of either an OS, hypervisor, or even aspects of the physical hardware. Typically, enclaves are also coupled with an attestation mechanism, which allows a remote party to ensure an enclave is running securely.
Historically, the open source software community rightfully was extremely wary of the technologies used to build enclaves. The technologies were primarily aimed to be applied as a way to prevent users from modifying their own computers and to enable DRM and has been criticized as antithetical to the philosophy of open source software. Today the technology has found a more useful place in the realms of privacy sensitive applications and safety-critical systems; the former is interesting because it inverts the model of DRM--instead of vendors not trusting end users with their devices, now it's the user's who do not trust the cloud vendor's servers.
| Technology | HW Arch | Resources, Examples | Key Mechanisms | Attestation Mechanism | Trusted Computing Base (TCB) | VM Enclave Support | Legacy App Support | Availability Support | Physical Defenses |
|---|---|---|---|---|---|---|---|---|---|
| Dedicated Processor Cores | * | Jailhouse, SecureCore | Use dedicated cores or a coprocessor to isolate VMs or OSes. Use either nested paging or bus security to protect physical memory. | Implementation defined. Likely would require a trusted hypervisor and bootloader. |
|
IMP | |||
| Virtualization + Trusted Boot | * | seKVM, pKVM, BlackBox | Using a hardware root-of-trust, a trusted "lowvisor" is loaded and authenticated. This lowvisor provides isolation of VMs or applications from an untrusted host OS through nested paging. | After trusted boot, lowvisor should generate attestation reports, though current implementations don't have this feature. |
|
||||
| Intel Software Guard Extensions (SGX) | x86 | Intel SGX Explained | All-in-one hardware extension for enclave segments of processes. Added instructions to the ISA (EENTER, ECREATE, etc.). Enclave page cache (EPC) is encrypted before being written to DRAM. | Hardware generates reports. Remote attestation uses Intel-provided provisioning and quoting enclaves. |
|
ME | |||
| Intel Trust Domain Extensions (TDX) | x86 | All-in-one hardware extension for VM enclaves protected against an untrusted hypervisor and environment. Added instructions to the ISA (SEAMCALL, etc.). Total memory encryption (TKTME) is used with different keys for VM Enclaves. | Hardware generates reports. Remote attestation uses Intel-provided provisioning and quoting enclaves. |
|
ME | ||||
| AMD Secure Encrypted Virtualization (SEV), SEV Encrypted State (SEV-ES), SEV Secure Nested Paging (SEV-SNP) | x86 | Terra, Flicker, TrustVisor | All-in-one hardware extension for VM enclaves protected against an untrusted hypervisor and environment. Adds a security coprocessor and additional ISA instructions for managing VM memory. MMU is extended with memory encryption engine which uses per-VM keys. Register states are encrypted on context switches. Integrity is enforced through a reverse map table. | Coprocessor generates signed report. Root of trust system used with AMD root CA at the beginning of the chain. |
|
ME | |||
| ARM TrustZone + Bus Security | ARMv7, ARMv8 | Pinto Survey, ARM Whitepaper | Hardware extension dividing processor into two security worlds. Additional EL3 privilege level added, and SMC instruction to call into EL3 mode. Programmable EL3 firmware (a.k.a. the Secure Monitor) and Secure world OS used to create enclaves/TEEs. Vendors must implement bus security, e.g. a TrustZone Address Space Controller (TZASC). Secure world may use interrupts to preempt normal world, and many system use a real-time OS as the secure-world OS. | Some research efforts have implemented remote attestation. |
|
IMP | |||
| ARM Confidential Compute Architecture (CCA) | ARMv9 | Li et al. | Hardware extension building on ARM TrustZone. Processor is further divided to add a Realm world. Programmable EL3 Secure Monitor context switches between the Normal world, Secure world, and Realm world. Realm world has a programmable (and verified) EL2 Realm Management Monitor (RMM) hypervisor to switch between realms. MMU is extended to support a Granule Partition Table (GPT) to protect physical memory. | No CCA specific mechanisms available yet. Likely implementations will attest Realm VM enclaves using a chain of trust from the Secure Monitor to RMM. |
|
IMP | |||
| Capability Hardware Enhanced RISC Instructions (CHERI) |
* | Introduction to CHERI, Project Oak Enclave | Generic ISA extension to add architectural capabilities. Memory accesses must be authorized using a capability held in a capability register file. Capabilities define the bounds a pointer may access and the permissions it may access. Valid pointers can only be derived from other pointers. A number of kernels and hypervisors have been derived from this technology. | No known mechanisms. |
|
||||
| RISC-V Machine Mode (M-Mode) | RISC-V | RISC-V ISA Manual, Keystone | RISC-V's physical memory protection (PMP) feature can create regions of isolated enclave memory. M-Mode can context switch between enclaves. | Some research efforts have implemented remote attestation. |
|
IMP | |||
| Pure Software Isolation | * | Virtual Ghost, RedLeaf | Language-based isolation and software-fault isolation techniques, including control-flow integrity (CFI), type safety, proof-carrying code (PCC), typed assembly (TAL), or formal verification may be useful for creating enclaves and some of these techniques have been explored in the past. | No known mechanisms. | Project specific, but often the compiler is part of the TCB. | ||||
| Fully Homomorphic Encryption & Verifiable Computation | * | Non-Interactive Verifiable Computing, Pinocchio | Fully Homomorphic encryption schemes allow computation on encrypted data while the agent performing the computation has no knowledge of the instructions or data being performed. Naturally this provides a similar protection as hardware-based enclave schemes. | Attestation is not needed, rather, a probabilistically checkable proof (PCP) is generated which can check that the computation was correctly performed. | Likely a circuit-garbling compiler will be trusted. | ME | |||
| Custom Processor Design/FPGA | * | AEGIS, Bastion, Sanctum | Outside of industry-provided hardware primitives, some academic proposals have demonstrated major and minor changes to hardware architectures to support enclave-like security. | Implementation specific | Implementation specific | ME |
| Key | |
|---|---|
| ME | Memory encryption. Bus snooping may reveal access patterns unless ORAM is used. |
| IMP | Implementation defined feature. |
| Support is unclear or unresearched. |
This table documents major technical contributions for enclave research in semi-chronological order. Incremental works by similar authors are grouped together as much as possible. C.I.A. in this table represent Confidentiality, Availability, and Integrity security properties respectively. RA indicates that remote attestation support is available, and P indicates that physical attacks like cold boot attacks and bus sniffing is protected against. MA stands for microarchitectural side channel attacks, like Spectre and Meltdown.
| Year(s) | Work | Venue(s) | Domain | Target Enclave | Key Technical Idea(s) | C | I | A | RA | P | MA | IO Support | Legacy |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2003 | AEGIS | ICS | Hardware | Process | Processor is extended to provide either memory integrity or memory encryption and integrity for enclave processes. Integrity is achieved through Merkle hash trees. | ||||||||
| 2003 | Terra | SOSP | Virt | VM | Uses a small trusted VMM to launch enclave VMs after a trusted boot. | ||||||||
| 2005 | Pioneer | SOSP | Software | Process | Using a nearly-perfectly optimized checksum function, this paper demonstrates how to build a pure software root of trust that allows a remote dispatcher to detect adversarial modifications by checking the execution time of the checksum. | ||||||||
| 2006 | Proxos | OSDI | Virt | Process | Design of a split OS enclave architecture using virtualization. One private/secure OS and one untrusted OS. Designs a language to allow an application to configure what system calls and paths in the filesystem are to be routed to what OS. | Ported | |||||||
| 2006 | Nizza | EuroSys | Virt | Process | Paravirtualizes L4Linux as a L4 process. Runs security sensitive applications (enclaves) as L4 tasks. | ||||||||
| 2008 | Overshadow, (follow up) | ASPLOS, HotSec | Virt | Process | Implements new concept of cloaking and multi-shadowing where each page of an enclave has an encrypted and unencrypted copy. The encrypted view is given to the untrusted kernel, and the unencrypted view is given to the user. Writes from kernel to encrypted page trigger decryption, and writes from enclave to plaintext trigger encryption. Provides a shim for communicating with the trusted VMM from the enclave, and handles many system calls. Follow up work discusses many of the problems with needing to trust the OS for system calls, and possible solutions. | ||||||||
| 2008 | SP3 | VEE | Virt | Process | Similar to Overshadow, a hypervisor uses a shadowed page table with encrypted and unencrypted copies of data selected dynamically depending on the executing security domain (user enclave or OS). Defines algorithm for lazy encryption and decryption to optimize performance. Hypervisor encrypts enclave register state on exception or interrupt and provides the OS an interface for switching into enclaves in order to decrypt that state. | ||||||||
| 2009 | Mobile Trusted Modules | CCSW/STC | TrustZone | Process | Presents essential techniques for using the TrustZone to run a secure and non-secure Linux instance, interrupt sharing, secure boot. Concept of a TrustZone-Assisted hypervisor is presented where the secure world and Monitor/EL3 mode is used as a hypervisor for the normal world. | ||||||||
| 2009 | Flicker | EuroSys | Virt | Segment | Uses AMD SEV or Intel TXT to execute a piece of app logic instead of a VM. On SKINIT/SENTER instruction, a loader saves the state of the OS, sets up a small execution environment for the application and jumps to ring 3. A TPM platform control register is updated up on completion to contain the attestation quote. | ||||||||
| 2010 | Bastion | HPCA | Hardware Virt | Segment | Hardware extensions supporting memory encryption and cache/register changes to track enclave memory. A new hypervisor is used to provide hypercalls which launch a small enclave segment/module process. Seems similar to AMD SEV + Flicker. | ||||||||
| 2010 | TrustVisor | IEEE S&P | Virt | Segment | Improves on Flicker by providing TMP services in software and only using hardware TPM functions when necessary. Removing the need to interact with a hardware TPM for basic enclave operations leads to a great performance improvement. | ||||||||
| 2010-2013 | SafeG | OSPERT, SWEST13, TRUST | TrustZone Virt | Process | Uses the ARM TrustZone to run a real-time OS. Interrupts are divided so the Secure world can preempt and schedule the normal world. Physical memory and devices are divided and isolated using TZASC. Real-time scheduler is aware of normal world scheduler and visa versa, they can collaborate to integrate scheduling. | Divided | RT | ||||||
| 2011 | CloudVisor | SOSP | Virt | VM | Deprivileges the Xen VMM into a guest VM, and runs Cloudvisor in host mode. Guests are run using nested virtualization; their page tables are stored in the VMM guest's address space, but in read-only mode. CloudVisor tracks page ownership and unmaps guest pages from the VMM. Cloudvisor interposes on VMEnter, VMExit and VMRead/VMWrite instructions as well as IOMMU faults. Disk IO is encrypted transparently. Some state is leaked to Xen to support IO. | Partial | |||||||
| 2012 | SMART | NDSS | Hardware | Generic | Demonstrates very minimal hardware extensions needed in order to establish a root of trust for an enclave. Using a challenge-response protocol, a remote verifier can dynamically attest a region of memory. The key needed create the response must be stored in a special ROM which is only readable by the SMART ROM code in charge of generating a response. Must be paired with other approaches to construct full enclaves. | Timing | |||||||
| 2013 | SecureCore | RTAS | Virt Multicore | Process | Runs a safety critical application on a dedicated core with it's own OS to provide availability for safety-critical processes. Uses nested paging and a custom hypervisor to provide memory and device isolation and communication. | Divided | |||||||
| 2013 | SASP | ICCVE | TrustZone Virt | Process | Designs support for shared devices using paravirtualization concepts applied to a TrustZone-assisted hypervisor. | Para | RT | ||||||
| 2013 | InkTag | ASPLOS | Virt | Process | Supports semi-unmodified Linux applications as enclaves with a subset of system call functionality. The InkTag hypervisor owns the real page tables for an enclave, and checks the OS's changes to its copies. The OS is allowed to change the page tables if it can produce a token which the enclave provided to it. This aims to capture the intent of the enclave to give access control to the page tables and prevent Iago attacks. | Semi | |||||||
| 2014 | Virtual Ghost | ASPLOS | Software | Process | The entire FreeBSD kernel is recompiled with a modified LLVM compiler which prevents binary code injection and ROP attacks by enforcing control-flow integrity (following up on previous work on "Secure Virtual Architectures"). This software isolation instrumentation is then used to implement ghost memory, which the OS cannot read or write, this is used to implement enclaves. The OS effectively needs to be ported to a new architecture to support the new compiler's handling of low-level state manipulation. | ||||||||
| 2014 | MiniBox | Usenix ATC | Virt | Process | Extends TrustVisor to support the execution of more rich applications (including legacy applications like OpenSSL) using Google's Native Client (NaCl). | Ported | |||||||
| 2014 | TrustLite | EuroSys | Hardware | Process | Hardware is proposed for small embedded devices without virtualization to extent the memory protection unit's (MPU) protection to include a column for code location. On a memory access, the table is indexed using the IP into the code column to determine whether the target address is accessible. Additional hardware is also needed to save register state of enclaves so leakage doesn't occur on interrupts. | Divided | |||||||
| 2014 | Trusted Language Runtime (TLR) | ASPLOS | TrustZone | Segment | Architecture for running managed .NET code in TrustZone. Secure world implements interpreter, type-system enforcer, and garbage collector. Uses proxy paradigm to perform RPC calls across worlds. | Partial | |||||||
| 2015 | SeCReT | NDSS | TrustZoneSoftware | Segment | Shows how a Linux process can do computation with a secret key that linux cannot read. When process tries to read the key it traps into EL3, which checks the integrity of the user process's CFG, the Link Register, and the integrity of the Linux vector trampolines, etc. If safe the key is loaded into the user process, and all traps are routed to EL3. Upon a second trap the key is scrubbed from the cache and memory. | ||||||||
| 2015 | TrustICE | ICDCS | TrustZone | Process | Simple approach to managing the TZASC on a system where only one Secure world memory range is configurable. Unfortunately, performance scales poorly with many enclaves due to copying. | ||||||||
| 2015 | `Haven | OSDI | SGX | Process | Uses a library OS to run unmodified Linux binaries on SGX. | ME | Recompile | ||||||
| 2016 | SCONE | OSDI | SGX | Container | Run unmodified Linux applications in a container in SGX. A way of performing system calls asynchronously is presented. | ME | Recompile | ||||||
| 2016 | Sanctum | Usenix Security | Hardware | Segment | An architecture for running enclaves with minimally invasive hardware changes. Provides a transparent implementation on RISC-V. Addresses previous cache side channel attacks in SGX by using a page coloring scheme to partition the cache. | ME | PC | ||||||
| 2016 | CaSE | IEEE S&P | TrustZone | Process | Supports running TrustZone applications purely in the cache and encrypting their data before being written to memory. NS flag is used to divide the cache and provide isolation on chip. Cache lines are loaded then locked into place, code cache lines must be clear to PoU (usually the LLC). | ME | |||||||
| 2016 | OSP Hypervisor | Usenix ATC | TrustZoneVirt | Process | Limits privilege escalation potential for enclaves by running them in the Normal world. While the Normal world is executing, TZASC protects enclave memory and single-level paging is used. While enclave is executing nested paging protects Normal and Secure world OSes. This is a performance optimization limiting the need for nested paging. | ||||||||
| 2016 | PrivateZone | TDSC | TrustZoneVirt | Segment | Similar to OSP Hypervisor, prevents potential for enclave privilege escalation by running enclaves in the Normal world. Difference is that the TZASC isn't dynamically adjusted and the normal world and enclaves always use nested page tables to enforce isolation. Android applications can be split into an enclave and non-enclave segment. | ||||||||
| 2017 | LTZVisor, LTZVisor-AMP | ECRTS, IECON | TrustZone | VM | Similar TZ-Assisted Hypervisor architecture to SafeG. Improves by implementing VirtIO like interface and supporting a multicore ARMv8 platform. | Divided | RT | ||||||
| 2017 | μRTZVisor | Electronics | TrustZone | VM | A design for a TZ-Assisted Hypervisor which is able to run multiple guest OSes. The TZASC is leveraged to isolate the currently running OS in the Normal World from the other OSes which are partitioned into the Secure World. Upon VM switch, the TZASC is configured to partition the memory so a different OS is in the Normal world. The cache and TLB must be completely flushed on switch. Secure devices are managed by EL3. | Divided | |||||||
| 2017 | Graphene-SGX | Usenix ATC | SGX | Process Group | Implements a library OS for Unmodified SGX applications. Achieves better performance than many other SGX designs showing feasibility of Library OS implementations. | ME | Partial | ||||||
| 2017 | TrustShadow | MobiSys | TrustZone | Process | Runs unmodified Linux applications in the secure world. System calls are forwarded by replaying the call in the vector table. Arguments are copied to the Normal world. Page tables are mostly managed by the secure world, table updates are communicated from Normal to Secure. | ||||||||
| 2017 | Komodo | SOSP | TrustZone | Segment | Implements a formally verified (using Dafny) SGX-like enclave interface using the ARM TrustZone. | ||||||||
| 2017 | vTZ | Usenix Security | TrustZoneVirt | Process | Supports virtualization of the ARM TrustZone and constrained enclave behaviors for better overall security. Ensures that the Normal-world Hypervisor cannot modify its own page tables by replacing all translation and exception configuring MRS/MSR instructions in its code and making the page tables and code sections and vector tables read only. The Secure world updates the page tables on behalf of the hypervisor. A region of Normal-world memory is then protected for use of creating virtualized enclaves. All exceptions are routed through the secure world by injecting SMCs into the vector table; this is done to support switching to and from enclaves (in addition to also replacing ERETs with SMCs). Enclaves must run to completion and interrupts are disabled during their execution, preventing them from DoSing the normal world. | ||||||||
| 2018 | RT-Trust | GPCE | TrustZone | Segment | Presents an automated way of splitting an real-time application into an enclave and non-enclave segments for TrustZone execution. | Refactored | |||||||
| 2019 | SecTEE | CCS | TrustZone | Segment | Implements an TrustZone enclave architecture which is resistant to side channel attacks and has a remote attestation scheme. Decrypts pages before they go off the SoC to protect against some physical attacks. | ME | PC | ||||||
| 2019 | HypSec | Usenix Security | Virt | VM | Similar to Cloudvisor, this paper splits the KVM hypervisor into two parts: a lowvisor and highvisor. Highvisor is deprivileged and run like a VM with nested paging. Lowvisor provides an API for interacting with creating and guests. Avoids CloudVisor's nested virtualization by introducing shadow nested page tables for the guest VMs, which are managed by the highvisor and shadowed by the lowvisor. Highvisor performs device emulation and IO paravirtualization. | ||||||||
| 2019 | Ginseng | NDSS | SoftwareVirt | Segment | C/C++ language and compiler are extended to be able to mark variables and functions as sensitive. Compiler propagates sensitive tag, disallows sensitive registers to state spill onto the stack. When state must be saved, compiler adds SMC or HVC calls to invoke a service to encrypt and save the variables onto a secure stack. OS's ability to control page tables removed and an API is provided to support/check page table updates. While secrets are in play/in registers, all exceptions route to EL3 by updating the EL1 vector tables. | Reg | |||||||
| 2019 | TEEv | VEE | TrustZoneVirt | VM | Prevents security risk of a vulnerable/malicious secure-world OS. Virtualizes a Secure-world OS without needing virtualization hardware. The Secure-world OS binary is scanned for any privileged instructions like MRS/MSR instructions, and they are replaced with calls into special gates which perform a trap. The S-EL1 page tables are setup for the Secure-world OS to prevent access to any other memory, including other Secure-world OSes or the Normal World. By locking the TLB entries for the gates globally across all ASIDs, the gate can safely swap the page tables from the Secure-world OS to the TEEv hypervisor. A few different IO paradigms are supported. | Divided | |||||||
| 2020 | Keystone | EuroSys | RISC-V | Process | Demonstrates how to use RISC-V's machine mode and phyiscal memory protection (PMP) system to create enclaves. | ME | PC | ||||||
| 2021 | TwinVisor | SOSP | TrustZone | VM | Runs a TrustZone Secure world VM via S-EL2. Secure world hypervisor lets EL1 manage page tables, and checks and syncs them with secure world. Develops a cooperative dynamic memory split. | ||||||||
| 2022 | RT-TEE | IEEE S&P | TrustZone | Process | Supports sharing devices between worlds using by replacing Normal world writes to their MMIO region with SMC calls. Policy of device usage is enforced by pattern matching the device access against a set of known access patterns. Implements availability in a similar way as SafeG and LTZVisor. | Sharing | |||||||
| 2022 | BlackBox | OSDI | Virt | Container | Similar to Flicker and TrustVisor, uses nested paging to isolate container enclaves. Container Security Monitor (CSM) at EL2 provides interface for OS to create container enclaves and manage page tables. All exceptions and interrupts pass through CSM by forcing the EL1 vector tables to jump to CSM. Supports unmodified applications by forwarding system calls and requiring collaboration between OS and CSM. | ||||||||
| 2022 | HyperEnclave | Usenix ATC | Virt | Segment | Runs (mostly) unmodified SGX applications using a hypervisor with nested paging. | ME |