Introducing PYCE: Executing Encrypted Python with No Performance Penalty
Soroco is excited to write its first blog post on how we modified the Python language to make deploying Python code more secure in production.
Though Python is a powerful and productive language for building systems, deploying it in production presents a large attack surface that allows a malicious user to modify or reverse engineer potentially sensitive business logic.
At Soroco, we always place protecting our client’s business logic first. This protection should be from both malicious attackers, and natural sources such as disk-based bit rot, or network transmission errors. Below, we describe how we encrypt and append an HMAC to each Python file that we ship for these purposes.
This project is open-source and available on GitHub. Further, you can get started with PYCE by downloading from PyPI:
Common Techniques and Challenges for Python
When we work with Python programs, there isn’t a single binary to sign. Unlike compiled languages, such as C, Rust, or Go, you don’t get a single executable to defend in the field.
With Python, your attack surface is much larger. So we can’t just sign a binary, obfuscation doesn’t actually protect things, and there are typically many files that would require encryption. Not to mention the fact that the Python interpreter does not have the capability of loading signed or encrypted files.
Thus, our requirements were threefold:
- Work with the reference implementation of Python,
- Provide strong protection of code against malicious and natural threats,
- Be performant both in execution time and in stored space.
This led us to a pure Python solution using authenticated cryptography.
Introducing the .pyce Extension
On our production servers, you’d find a peculiar file extension:
.pyce. You’re probably used to
.pyc—compiled Python bytecode—and
.py—raw Python source. But you’ve never seen
.pyce before. That’s because it only exists at Soroco!
Figure 1. The
.pyce on disk format.
At runtime, we verify the HMAC before decrypting the ciphertext. This ensures the integrity—each bit is exactly as intended—of the code as well as its authenticity—it came from an entity controlling the secret key. It also ensures that we waste no time decrypting or executing even a single bit that was not intended by our engineers.
Loading .pyce in Python
But how do we get a reference implementation
Python.exe running the
It turns out Python’s interpreter has import machinery that you can hook to load whatever you want. In essence, you can customize and change the meaning of
import in Python with ease!
Figure 2. We implemented loading
.pyce by hooking into
sys.meta_path and inheriting from pre-defined import machinery classes.
The above Figure 2 illustrates how we hook into the Python import machinery. We add a special
SorocoPathFinder at the beginning of the list of known module loaders.
sys.meta_path lists all loaders for the Python interpreter. Three shown above should be familiar:
How does the code work? Primarily, we reuse the two intermediate classes for their pre-existing implementation:
FileFinder. In other words, we benefit from all of the engineering already implemented in the import machinery. These two classes search the Python path and only call our special decrypting loader if they find a module file with the right extension:
.pyce. All of the magic happens within the
SorocoFileLoader which inherits from
Figure 3. Code from the
SorocoFileLoader class demonstrating the decrypting and loading of an encrypted file.
The code above is largely taken from the Python reference implementation, with a single line inserted calling a function called
decrypt. The fact that its implementation is just 5 lines of executable code is a testament to the flexibility of the Python language.
It is precisely this powerful flexibility that Soroco leverages in its proprietary Python runtime focused on business process automation.
Optimizing Space with Convergent Encryption
Across our production systems, we often deploy a lot of duplicate files. This is due to shared libraries, in some cases shared codebases, and shared resources. As an optimization, we deduplicate our storage layers to minimize our disk footprint.
But how can we deduplicate encrypted files? Encrypted data is, by design, indistinguishable from random data. Thus, if we encrypted the same Python file 10 times, we would produce 10 unique files.
We use convergent encryption to protect sensitive business logic, but still produce the same ciphertext when presented with the same plaintext. We are in a scenario such that attackers can not choose plaintext. Therefore, they can only gain information about publicly known files—such as open source libraries. Since we already reveal the open source libraries we use in production, we do not consider this as a risk at all.
Figure 4. Convergent encryption implementation using AES-256 and SHA-256.
In Figure 4 above, we reproduce our core convergent encryption code. Convergent encryption fundamentally depends on deterministic generation of an encryption key. We use SHA-256 to deterministically generate a unique 256-bit key for each file, which we then encrypt with AES-256. The nonce for counter mode is deterministically set to the first 16 bytes of the key. After encrypting, we use an HMAC with SHA-512 as its hash algorithm, keyed deterministically with the same SHA-256 hash of the plaintext.
Naturally, we never want the keys to touch the disk in a plaintext format. We always store keys in a secure storage layer implementing a cryptographic barrier—meaning data at rest is always encrypted. For example, Hashicorp’s Vault implements a secure write barrier. When we execute on production servers, our runtime pipes in a JSON formatted dictionary of all of the keys via
stdin to the executing Python process. It then resumes normal execution after reading in the keys. Whenever an import triggers loading a
.pyce file, our custom
SorocoFileLoader, as shown in Figure 2, checks this cache of keys, verifies the HMAC, decrypts, and loads the bytecode into Python’s in-memory cache.
Performance and Impact to Production
Our implementation has no overhead in production. This is due to Python’s in-memory bytecode cache. Only when our processes begin execution, and first import bytecode, does decryption take place. Thus, during normal operation there is no impact to the runtime of Soroco business logic!
For the very paranoid, you should pin your Python process’s memory space such that it never pages out to disk. Pinning memory guarantees that plaintext business logic will never be persistently stored, and neither will the decryption keys. Due to Python’s in-memory bytecode cache, decrypted business logic could get paged out to disk. Pinning your process’s memory space eliminates this risk.
Working with Python’s import machinery was easier than we expected. Our entire implementation is less than 150 lines of Python code, with no performance impact to running business logic in production. Such a compact codebase makes auditing much easier. And we are fully open sourcing it here: Soroco PYCE.
Like this article? Spread the word