I'd like to see a formal container security grade that works like:
1) Curate a list of all known (container) exploits
2) Run each exploit in environments of increasing security like permissions-based, jail, Docker and emulator
3) The percentage of prevented exploits would be the score from 0-100%
Under this scheme, I'd expect naive attempts at containerization with permissions and jails to score around 0%, while Docker might be above 50% and Microsandbox could potentially reach 100%.
This might satisfy some of our intuition around questions like "why not just use a jail?". Also the containers could run on a site on the open web as honeypots with cash or crypto prizes for pwning them to "prove" which containers achieve 100%.
We might also need to redefine what "secure" means, since exploits like Rowhammer and Spectre may make nearly all conventional and cloud computing insecure. Or maybe it's a moving target, like how 64 bit encryption might have once been considered secure but now we need 128 bit or higher.
Edit: the motivation behind this would be to find a container that's 100% secure without emulation, for performance and cost-savings benefits, as well as gaining insights into how to secure operating systems by containerizing their various services.
tptacek · 9h ago
The issue, at least with multitenant workloads, isn't "container vulnerabilities" as such; it's that standard containers are premised on sharing a kernel, which makes every kernel LPE a potential container escape --- there's a long history of those bugs, and they're only rarely flagged as "container escapes"; it's just sort of understood that a kernel LPE is going to break containers.
bjackman · 9h ago
You cannot build a secure container runtime (against malicious containers) because underlying it is the Linux kernel.
The only way to make Linux containers a meaningful sandbox is to drastically restrict the syscall API surface available to the sandboxee, which quickly reduces its value. It's no longer a "generic platform that you can throw any workload onto" but instead a bespoke thing that needs to be tuned and reconfigured for every usecase.
This is why you need virtualization. Until we have a properly hardened and memory safe OS, it's the only way. And if we do build such an OS it's unclear to me whether it will be faster than running MicroVMs on a Linux host.
akdev1l · 9h ago
One can definitely build a container runtime that uses virtualization to protect the host
This can be used with regular `podman` by just changing the container runtime so there’s no even need for any extra tooling
In theory you could shove the container runtime into something like k8s
Veserv · 9h ago
You cannot build a secure virtualization runtime because underlying it is the VMM. Until you have a secure VMM you are subject to precisely the same class of problems plaguing container runtimes.
The only meaningful difference is that Linux containers target partitioning Linux kernel services which is a shared-by-default/default-allow environment that was never designed for and has never achieved meaningful security. The number of vulnerabilities resulting from, "whoopsie, we forgot to partition shared service 123" would be hilarious if it were not a complete lapse of security engineering in a product people are convinced is adequate for security-critical applications.
Present a vulnerability assessment demonstrating a team of 10 with 3 years time (~10-30 M$, comparable to many commercially-motivated single-victim attacks these days) can find no vulnerabilities in your deployment or a formal proof of security and correctness otherwise we should stick with the default assumption that software if easily hacked instead of the extraordinary claim that demands extraordinary evidence.
transpute · 4h ago
> You cannot build a secure virtualization runtime because underlying it is the VMM
There are VMMs (e.g. pKVM in upstream Linux) with small SLoC that are isolated by silicon support for nested virtualization. This can be found on recent Google Pixel phones/tablets with strong isolation of untrusted Debian Arm Linux "Terminal" VM.
A similar architecture was shipped a decade ago by Bromium and now on millions of HP business laptops, including hypervisor isolation of firmware, "Hypervisor Security : Lessons Learned — Ian Pratt, Bromium — Platform Security Summit 2018", https://www.youtube.com/watch?v=bNVe2y34dnM
While VMs do have an attack surface, it is vastly different than containers, which as you pointed out are not really a security system, but simply namespaces.
Seacomp, capabilities, selinux, apparmor, etc.. can help harden containers, but most of the popular containers don't even drop root for services, and I was one of the people who tried to even get Docker/Moby etc.. to let you disable the privileged flag...which they refused to do.
While some CRIs make this easier, any agent that can spin up a container should be considered a super user.
With the docker --privlaged flag I could read the hosts root volume or even install efi bios files just using mknod etc, walking /sys to find the major/minor numbers.
Namespaces are useful in a comprehensive security plan, but as you mentioned, they are not jails.
It is true that both VMs and containers have attack surfaces, but the size of the attack surface on containers is much larger.
ignoramous · 9h ago
> ... drastically restrict the syscall API surface available to the sandboxee, which quickly reduces its value ...
Depends I guess as Android has had quite a bit of success with seccomp-bpf & Android-specific flavour of SELinux [0]
> Until we have a properly hardened and memory safe OS ... faster than running MicroVMs on a Linux host.
Andy Tanenbaum might say, Micro Kernels would do just as well.
You also have gVisor, which runs all syscall through some Go history that's supposedly safe enough for Google.
godelski · 9h ago
Importantly I'd like to see the configurations of the machines. There's a lot you can do to docker or systemd spawns that greatly vary the security levels. This would really help show what needs to be done and what configurations lead to what risks.
Basically I'd love to see a giant ablation
Etheryte · 9h ago
In a way, containers already run as honeypots with cash or crypto prizes, it's called production code and plenty of people are looking for holes day and night. While this setup sounds like a nice idea conceptually, the monetary incentives it could offer would surely be miniscule compared to real targets.
dataflow · 14h ago
Tangential question: why does it normally take so long to start traditional VMs in the first place? At least on Windows, if you start a traditional VM, it takes several seconds for it to start running anything.
Edit: when I say anything, I'm not talking user programs. I mean as in, before even the first instruction of the firmware -- before even the virtual disk file is zeroed out, in cases where it needs to be. You literally can't pause the VM during this interval because the window hasn't even popped up yet, and even when it has, you still can't for a while because it literally hasn't started running anything. So the kernel and even firmware initialization slowness are entirely irrelevant to my question.
Why is that?
jeroenhd · 13h ago
You can optimize a lot to start a Linux kernel in under a second, but if you're using a standard kernel, there are all manners of timeouts and poll attempts that make the kernel waste time booting. There's also a non-trivial amount of time the VM spends in the UEFI/CSM system preparing the virtual hardware and initializing the system environment for your bootloader. I'm pretty sure WSL2 uses a special kernel to avoid the unnecessary overhead.
You also need to start OS services, configure filesystems, prepare caches, configure networking, and so on. If you're not booting UKIs or similar tools, you'll also be loading a bootloader, then loading an initramfs into memory, then loading the main OS and starting the services you actually need, with eachsstep requiring certain daemons and hardware probes to work correctly.
There are tools to fix this problem. Amazon's Firecracker can start a Linux VM in a time similar to that of a container (milliseconds) by basically storing the initialized state of the VM and loading that into memory instead of actually performing a real boot. https://firecracker-microvm.github.io/
On Windows, I think it depends on the hypervisor you use. Hyper V has a pretty slow UEFI environment, its hard disk access always seems rather slow to me, and most Linux distro don't seem to package dedicated minimal kernels for it.
dataflow · 12h ago
That's not what I'm asking about.
I'm saying it takes a long time for it to even execute a single instruction, in the BIOS itself. Even for the window to pop up, before you can even pause the VM (because it hasn't even started yet). What you're describing comes after all that, which I already understand and am not asking about.
hnuser123456 · 12h ago
probably the intel ME setting up for virtualization in a way that it can infiltrate
LoganDark · 11h ago
Ah yes, the source of all slowness in the CPU: hostile backdoors taking their time to compromise the work. Classic...
orev · 13h ago
I think you need to provide more details on what VM software you’re using. On VirtualBox what you describe is very noticeable, and it didn’t have that delay in older versions. So it could be just an issue with that VM software and not a general “traditional VMs” issue.
dataflow · 13h ago
Yup I'm asking about VirtualBox mainly, I just don't understand what the heck it's doing during that time that takes so long. Although I don't recall other VMs (like say, Hyper-V) being dramatically different either (ignoring WSL2 here).
icedchai · 6h ago
Linux KVM/qemu VMs start pretty fast.
_factor · 13h ago
Try disabling Windows Defender and trying again.
dataflow · 12h ago
Are you just guessing or have you actually seen the delay I'm talking about disappear as a result of this (or as a result of anything else for that matter)? Because I've already done this (yes, entirely, even the kernel mode drivers) and it's definitely not the issue.
hinkley · 12h ago
There was a release of subversion back in the day that reduced the number of files that were opened during a repo action like pull, and the number of times any one file got opened. On Linux it ran about 2-3x faster. Very nice change.
On windows it was almost 10x faster. On the project where this change was released, my morning ritual was to come in, log on, run an svn pull command, lock my screen and go get coffee. I had at least ten minutes to kill after I got coffee, if the pot wasn’t empty when I got there.
Windows is hot garbage about fopen particularly when virus scanning is on.
akdev1l · 9h ago
The answer is that it doesn’t have to be like that.
In practice virtual machines are trying to emulate a lot of stuff that isn’t really needed but they’re doing it for compatibility.
If one builds a hypervisor which is optimized for startup speed and doesn’t need to support generalized legacy software then you can:
> Unlike traditional VMs that might take several seconds to start, Firecracker VMs can boot up in as little as 125ms.
BobbyTables2 · 5h ago
In Linux, VM memory allocations can be slow if it tries to allocate GBs of RAM using 4K pages. There are ways to help it allocate 1GB at a time which vastly speeds it up.
Windows probably has an equivalent.
dist-epoch · 11h ago
Sounds like a VirtualBox problem.
I'm using Hyper-V and I can connect through XRDP to a GUI Ubuntu 22 in 10 seconds and I can SSH into a Ubuntu 22 server in 3 seconds after start.
jiggawatts · 6h ago
Try Windows Server Core on an SSD. I've seen VMs launch in low single-digit seconds. You can strip it down even further by removing non-64-bit support, Defender, etc...
speed_spread · 12h ago
Creating the VM itself is fast. It depends on what you run in it. Unikernel VMs can start in a few milliseconds. For example, checkout OSv.
dataflow · 12h ago
You're saying this is true on a Windows host?
akdev1l · 8h ago
Yes. The delay you’re complaining about happens because you are looking at general hypervisors which also come with virtualized hardware and need to mimic a bunch of stuff so that most software will work as usual.
For example: your VM starts up with the CPU in 16 bit mode because that’s just how things work in x86 and then it waits for the guest OS to set the CPU into 64 bit mode.
This is completely unnecessary if you just want to run x86-64 code in a virtualized environment and you control the guest kernel and can just assume things are in 64bit mode because it’s not the 70s or whatever
The guest OS would also need to probe few ports to get a bootable disk. If you control the kernel then you can just not do that and boot directly.
There’s a ton of stuff that isn’t needed
dataflow · 8h ago
The 16 bit mode stuff and the guest OS probes are after what I'm asking, not before.
akdev1l · 8h ago
No it is not. The “first instruction in the BIOS” is 16 bit mode code when dealing with an x86 VM.
A virtual environment doesn’t even really need any BIOS or anything like that.
You can feel free to test with qemu direct kernel booting to see this skips a lot of delay without even having to use a specialized hypervisor like firecracker
diggan · 13h ago
I mean it is basically booting a computer from scratch, kind of makes sense. You have to allocate memory, start virtual CPUs, initialize devices, run BIOS/UEFI checks, perform hardware enumeration, all that jazz while emulating all of it, which tends to be slower than "real" implementations. I guess there is a bunch of processes for security as well, like wiping like zeroing pages and similar things that takes additional time.
If I let a VM use most of my hardware, it takes a few seconds from start to login prompt, which is the same time it takes for my Arch desktop to boot from pressing the button to seeing the login prompt.
dataflow · 13h ago
> You have to allocate memory, start virtual CPUs, initialize devices, run BIOS/UEFI checks, perform hardware enumeration, all that jazz while emulating all of it, which tends to be slower than "real" implementations.
That's not what I'm asking.
I'm saying it takes a long time for it to even execute a single instruction, in the BIOS itself. Even for the window to pop up, before you can even pause the VM (because it hasn't even started yet). What you're describing comes after all that, which I already understand and am not asking about.
bityard · 12h ago
In defense of the replies, your initial question was very vague and left people to assume you meant the obvious thing.
dataflow · 12h ago
Sure, that's why I clarified.
drewg123 · 12h ago
Without any context in terms of what the VM is doing or what VMM software you use, my best guess is that the OS/VMM are pre-allocating memory for the VM. This might involve paging out other processes' memory, which could take some time.
I think task manager would tell you if there is a blip of memory usage and paging activity at the time. And I'm sure windows itself has profilers that can tell you what is happening when the VM is started..
dataflow · 12h ago
VirtualBox on Windows, primarily. Though I feel like haven't seen other VMs in the past start up a whole ton faster (maybe a somewhat) (ignoring WSL2). Page files are already disabled, there's plenty of free RAM, and it makes no difference how little RAM the guest is allocated (even if it's 256MB). So no, those are not the issues. VirtualBox itself seems to be doing something slow during that time and I don't know what that is.
gopher_space · 8h ago
I remembered something about VirtualBox not playing nicely with Hyper-V on Windows, and dug up a possibly relevant post[0] on their forums. IIRC we ended up moving a few build systems to Docker and dropping VirtualBox because of hyper-v related issues, but it's been a few years.
That's the unrelated green-turtle issue. It's only relevant after the guest has actually started running instructions. I'm talking about before that point.
gopher_space · 7h ago
I'm not aware of any turtles, that was just the first thing I found when trying to see if VirtualBox and Hyper-V were still a problematic combo.
Again, it was a few years ago, but we didn't solve the problem or identify an actual root cause. We stopped banging our heads against that particular wall and switched technologies.
mynameisvlad · 10h ago
So the issue is pretty clearly with VirtualBox itself, but you are making it sound like it's an issue with VMs on Windows or in general.
drewg123 · 12h ago
For some reason I can't reply to your reply. I'd strongly suggest that you profile virtual box. It beats speculation..
eamann · 8h ago
> Ever needed to run code you don't fully trust?
Then the installation instructions include piping a remote script directly to Bash ... Oh irony ...
That said, the concept itself is intriguing.
appcypher · 7h ago
Your statement initially went over my head. Sorry lol.
You can always download the installer script and audit yourself.
I will set up proper distribution later.
hakcermani · 4h ago
.. did exactly that and also changed the BINDIR and LIBDIR to another location.
BTW, amazing project from initial glance. Will give it a detailed look this weekend!
appcypher · 15h ago
Thanks for sharing!
I'm the creator of microsandbox. If there is anything you need to know about the project, let me know.
This project is meant to make creating microvms from your machine as easy as using Docker containers.
Ask me anything.
spicybright · 12m ago
I like the idea. But when you say "bullet proof" security, there are exploits to break out of VMs that exist. Have you looked into those?
nulld3v · 29m ago
Cool project. Off topic question: Are the images in the "Use Cases" section in the README from a real app? I like the clean UI design.
simonw · 9h ago
I'm trying this out now and it's very promising. One problem I'm running into with the Python library is that I'd like to keep that sandbox running for several minutes while I do things like set variables in one call and then use them for stuff several calls later. I keep seeing this error intermittently:
Error: Sandbox is not started. Call start() first
Is there a suggested way of keeping a sandbox around for longer?
The documented code pattern is this:
async def main():
async with PythonSandbox.create(name="my-sandbox") as sb:
exec = await sb.run("print('Hello, World!')")
print(await exec.output())
Due to the way my code works I want to instantiate the sandbox once for a specific class and then have multiple calls to it by class methods, which isn't a clean fit for that "async with" pattern.
Any recommendations?
appcypher · 7h ago
Right. You can skip the `with` context manager and call start and stop yourself.
async with is just syntactic sugar. You could very well call __aenter__ and __aexit__ manually. You could also use an AsyncExitStack, call __aenter__ manually, then enter_async_context, and call aclose when you’re done. Since aclose method exists I guess this is not an anti-pattern.
Looks great! This might be extremely useful for a distributed/decentralized software testing network I'm building (called Valet Network)...
Question: How does networking work? Can I restrict/limit microvms so that they can only access public IP addresses? (or in other words... making sure the microvms can't access any local network IP addresses)
thanks! have an example on how to use that in a sandboxfile?
(also, this project is really cool. great work!)
appcypher · 10h ago
Yeah. I need to fix that in the docs!
hugs · 10h ago
no prob!
codethief · 6h ago
Hi appcypher, very cool project! Does the underlying MicroVM feature provide an OCI runtime interface, so that it could be used as a replacement for runc/crun in Docker/Podman?
Nypro · 6h ago
No. Not yet. Would be nice to have
catlifeonmars · 1h ago
How does the microvm architecture compare with firecracker?
0cf8612b2e1e · 14h ago
Only did a quick skim of the readme, but a few questions which I would like some elaboration.
How is it so fast? Is it making any trade offs vs a traditional VM? Is there potential the VM isolation is compromised?
Can I run a GUI inside of it?
Do you think of this as a new Vagrant?
How do I get data in/out?
appcypher · 14h ago
> How is it so fast? Is it making any trade offs vs a traditional VM? Is there potential the VM isolation is compromised?
It is a lighweight VM and uses the same technology as Firecracker
> Can I run a GUI inside of it?
It is planned but not yet implemented. But it is absolutely possible.
> Do you think of this as a new Vagrant?
I would consider Docker for VMs instead. In a similar way, it focuses on dev ops type use case like deplying apps, etc.
> How do I get data in/out?
There is an SDK and server that help does that and file streaming is planned. But right now, you can execute commands in the VM and get the result back via the server
> AWS built [Firecracker (which is built on KVM)] to power Lambda and Fargate [2], where they need to quickly spin up isolated environments for running customer code. Companies like E2B use Firecracker to run AI generated code securily in the cloud, while Fly.io uses it to run lightweight container-like VMs at the edge [4, 5].
Yes. Self-hosting and using it on your own backend infra is the main use-case.
And JVM support should just work since it is a Linux machine.
nqzero · 11h ago
i'm on a mid-level laptop, at times with slow or expensive internet, running ubuntu. i want to be able to run nominally-isolated "copies" of my laptop at near-native speed
1. each one should have it's own network config, eg so i can use wireguard or a vpn
2. gui pass-through to the host, eg wayland, for trusted tools, eg firefox, zoom or citrix
3. needs to be lightweight. eg gnome-boxes is dead simple to setup and run and it works, but the resource usage was noticeably higher than native
4. optional - more security is better (ie, i might run semi-untrusted software in one of them, eg from a github repo or npm), but i'm not expecting miracles and accept that escape is possible
5. optional - sharing disk with the host via COW would be nice, so i'd only need to install the env-specific packages, not the full OS
i'm currently working on a podman solution, and i believe that it will work (but rebuilding seems to hammer the network - i'm hoping i can tweak the layers to reduce this). does microsandbox offer any advantages for this use case ?
appcypher · 11h ago
> 1. each one should have it's own network config, eg so i can use wireguard or a vpn
This is possible right now but the networking is not where I want it to be yet. It uses libkrun's default TSI impl; performant and simplifies setup but can be inflexible. I plan to implement an alternative user-space networking stack soon.
> 2. gui pass-through to the host, eg wayland, for trusted tools, eg firefox, zoom or citrix
We don't have GUI passthrough. VNC?
> 3. needs to be lightweight. eg gnome-boxes is dead simple to setup and run and it works, but the resource usage was noticeably higher than native
It is lightweight in the sense that it is not a full vm
> 4. optional - more security is better (ie, i might run semi-untrusted software in one of them, eg from a github repo or npm), but i'm not expecting miracles and accept that escape is possible
The security guarantees are similar to what typical VMs support. It is hardware-virtualized so I would say you should be fine.
> 5. optional - sharing disk with the host via COW would be nice, so i'd only need to install the env-specific packages, not the full OS
Yeah. It uses virtio-fs and has overlayfs on top of that for COW.
Hilift · 5h ago
Are you ready for the deluge of networking questions for all the buck wild configurations?
Nypro · 5h ago
Lol. I should brace for impact.
Networking continues to be a pain but I'm open to suggestions.
wolfhumble · 11h ago
Can you use Microsandbox for everything you can use Docker for, or are there cases where containers make more sense?
Congratulations on the launch!
appcypher · 10h ago
We want microsandbox to be usable for everything you can with Docker.
That said, hosting microVMs require dedicated hardware or VMs with nested virt support. Containers don’t have that problem.
simonw · 13h ago
What's the story for macOS support?
appcypher · 13h ago
It uses libkrun which uses Hypervisor.framework on macOS.
McAlpine5892 · 13h ago
This looks awesome. The amount of super lightweight and almost-disposable VM options in recent years is crazy. I remember when VMs were slow, clunky, and generally painful.
I wonder how this compares to Orbstack's [0] tech stack on macOS, specifically the "Linux machines" [1] feature. Seems like Orb might reuse a single VM?
Congrats on launching! Booting VMs in milliseconds is certainly important, but it can also be achieved with CloudHypervisor/Firecracker. Where Containers beat VMs is runtime perf. The overhead in case of VMs stems from emulation of IO devices. I believe the overhead will become noticeable for AI agentic use cases. Any plans to address perf issues?
appcypher · 11h ago
You are right. We leverage libkrun. Libkrun uses virtio-mmio transport for block, vsock and virtio-fs to keep overhead minimal so we basically depend on any perf improvement made upstream.
Firecracker is no different btw and E2B uses that for agentic AI workloads. Anyway, I don't have any major plan except fix some issues with the filesystem rn.
amelius · 7h ago
For my taste, container technology is pushing the OS too far. By typing:
mount
you immediately see what I mean. Stuff that should be hidden is now in plain sight, and destroys the usefulness of simple system commands. And worse, the user can fiddle with the data structures. It's like giving the user peek and poke commands.
The idea of containers is nice, but they are a hack until kernels are re-architected.
throwaway314155 · 6h ago
Sorry I am lacking the context to understand this post. What does running mount inside a container do that's so egregious? Are host mounts exposed to the container somehow? I thought everything needed to be explicitly passed through to the container (e.g. using a volume)?
rbitar · 10h ago
Looks great and excited to try this out. We’ve also had success using CodeSandbox SDK and E2B, can you share some thoughts on how you compare or future direction? Do you also use Firecracker under the hood?
appcypher · 9h ago
> can you share some thoughts on how you compare or future direction?
Microsandbox does not offer a cloud solution. It is self-hosted, designed to do what E2B does, to make it easier working with microVM-based sandboxes on your local machine whether that is Linux, macOS or Windows (planned) and to seamlessly transition to prod.
> Do you also use Firecracker under the hood?
It uses libkrun.
rbitar · 9h ago
Self-hosting is definitely something we are keen to explore as most of the cloud solutions have resource constrains (ie, total active MicroVMs and/or specs per VM) and managing billing gets complicated even with hibernation features. Great project and we'll definitely take it for a spin
pkkkzip · 10h ago
I can't tell if it uses firecracker but thats my main question too. I'm curious as to whether microsandbox will be maintained and proper auditing will be done.
I welcome alternatives. It's been tough wrestling with Firecracker and OCI images. Kata container is also tough.
appcypher · 9h ago
It will be maintained as I will be using it for some other product.
And it will be audited in the future but it still early days.
sureglymop · 10h ago
Always interested when things like this come up.
What like about containers is how quickly I can run something, e.g. `docker run --rm ...` without having to specify disk size, amount of cpu cores, etc. I can then diff the state of the container with the image (and other things) to see what some program did while it ran.
So I basically want the same but instead with small vms to have better sandboxing. Sometimes I also use bwrap but it's not really intended to be used on the command line like that.
srmatto · 9h ago
It has a YAML config format to declare all of that so you could just do that once, or template it, generate it on the fly, fetch it from remote, or many other methods.
SwiftyBug · 11h ago
Kind of almost off-topic: I'm working on a project where I must run possibly untrusted JavaScript code. I want to run it in an isolated environment. This looks like a very nice solution as I could spin up a microsandbox and securely run the code. I could even have a pool os live sandboxes so I wouldn't even experience the 200ms starts. Because this is OCI-compatible, I could even provide a whole sandboxed environment on which to run that code. Would that be a good use case for this? Are there better alternatives?
runsc / gVisor is interesting also as the runsc engine can be run from within Docker/Docker Desktop.
gVisor has performance problems, though. Their data shows 1/3rd the throughput vs. docker runtime for concurrent network calls--if that's an issue for your use-case.
appcypher · 11h ago
> Would that be a good use case for this?
That is an ideal use case
> Are there better alternatives?
Created microsandbox because I didn't find any
SwiftyBug · 11h ago
Awesome. This is really good timing. I'm going to give it a try.
Jayakumark · 14h ago
Windows support ? and can we VNC in to the sandbox and stream it ?
appcypher · 14h ago
Windows support is a work in progress.
I haven't tested using VNC yet but it should be possible.
> Can we build our own python sandbox using the sandboxfile spec?
Yes and I plan to make that work with the SDK.
PS: Multi-stage build is WIP.
Tsarp · 12h ago
Great will join the discord. Is this embeddable? Will it work with a cross platform desktop app(Tauri)?
apitman · 1h ago
An embeddable library that lets you launch Linux VMs that works across Windows, MacOS, and Linux hosts would be incredible.
sbassi · 7h ago
There are python and node environment for this, so they are not VMs in the sense that I can host a OS and arbitrary executables?
appcypher · 7h ago
They are Linux VMs and you can host any executable that can work on that.
The python/node environment you see is part of what makes the SDK work. Really, it's very similar to Docker in use.
sbassi · 7h ago
thank you. Is there any "docker host" or centralized repo where I can pull VMs from?
appcypher · 7h ago
We support just Docker hub for now. Let me know if you want any other OCI-compatible registry.
PS: microsandbox will likely have its own OCI registry in the future
elwebmaster · 11h ago
One topic I am not finding anything about is networking. Can these microsandbox instances listen on ports? How is the port forwarding configured? Can they access the internet or any resources on the host?
appcypher · 10h ago
They can. I need to improve the doc. Working on that right now
manveru · 12h ago
Are the SDKs AI generated? I looked at the Crystal, Ruby, and Zig ones and all they contain is a hello world example with some docs that have little to do with the code.
Sorry if this comment seems rude, just curious.
appcypher · 12h ago
The other SDKs are generated hello-worlds at the moment. I will get to them one by one, but I welcome and appreciate any contributions to them.
jmehman · 12h ago
I've been looking for something I could host for this kind of thing - for LLM agents. Ended up on https://www.daytona.io/ as I couldn't find anything suitable to self host and realised it was a complex thing to manage. It seems Daytona is open source, including the server platform, but there is no documentation for the server element. Azure also seem to offer a service for this, it's a space that is growing rapidly.
appcypher · 12h ago
Microsandbox is for people that would like to maintain their own infra.
I'm not going to stop trying to make it better to self-host.
jmehman · 12h ago
Yeah, it looks great, makes me reconsider the self hosted route
h1fra · 13h ago
Can't wait to test, if it's really what's advertised it would be much easier to use than workerd or firecracker
patrick4urcloud · 12h ago
very nice !
i will definetly try
hinkley · 11h ago
How’s performance? What’s the overhead versus docker? Terraform or Pulumi integration on the horizon?
The orientation as a safe sandbox for AI / MCP tools is a very nicely packaged looking experience, and very well marketred. Congratulations! I'm still not sure why this warrants being it's own project.
simonw · 14h ago
If we get enough of these sandboxes, maybe we will finally get one that's easy for me to run on my own machines.
hobofan · 12h ago
Exactly my thoughts when I read the headline, after having read a similar one every few months.
However, by looking at it and playing with a few simple examples, I think this is the one that looks the closest so far.
Definitely interested to see the FS support, and also some instruction on how to customize the images to e.g. pre-install common Python packages or Rust crates. As an example, I tried to use the MCP with some very typical use-cases for code-execution that OpenAI/Anthropic models would generate for data analysis, and they almost always include using numpy or a excel library, so you very quicly hit a wall here without the ability to include libraries.
appcypher · 13h ago
That's the plan lol.
There is too much friction setting up existing solutions.
tough · 14h ago
would you be OK with a -hardened- with default profiles docker containers one?
appcypher · 13h ago
I don't understand what you mean? Can you clarify?
tough · 13h ago
sorry i meant to ask simon directly if they require a non-docker solution
im working on a wrapper that lets you swap runtimes and my first implementation is mostly a wrapper around docker containers
planning to add firecracker next
will explore adding microsandbox too cool stuff!
simonw · 10h ago
My ideal solution is non-Docker purely because I build software for other people to use. I don't want to have to tell my users "step 1: install Docker" if I can avoid it.
tough · 9h ago
that does make sense, sadly firecracker seems to be mostly relegated to linux for now so there's no good multi-arch story i'm aware of
mike_hearn · 14h ago
Which platforms do you use?
simonw · 13h ago
macOS on my laptop, anything that runs in a container for when I deploy things.
I had luck using ALVM which users Apple Hypervisor framework while exploring linux micro-vm's in macos fwiw https://github.com/mathetake/alvm
simonw · 10h ago
That looks really cool, but it's missing the one feature I want most from anything that runs a sandbox (or any security-related software): I need something which a billion dollar company with a professional security team is running in production on a daily basis.
So much of the solutions to this stuff I see come from a GitHub repo with a few dozen commits and often a README that says "do not rely on this software yet".
Definitely going to play with it a bit though, I love the idea of hooking into Apple's Hypervisor.framework (which absolutely fits my billion-dollar-company requirement.)
appcypher · 14h ago
Because those have different directions than microsandbox and you've already mentioned one. I want easy secure sandboxes for AI builders. IMHO, microsandbox is easier to get started with.
That said I don't think either KataContainer or Cloud Hypervisor has first-class support for macOS.
I'd like to see a formal container security grade that works like:
Under this scheme, I'd expect naive attempts at containerization with permissions and jails to score around 0%, while Docker might be above 50% and Microsandbox could potentially reach 100%.This might satisfy some of our intuition around questions like "why not just use a jail?". Also the containers could run on a site on the open web as honeypots with cash or crypto prizes for pwning them to "prove" which containers achieve 100%.
We might also need to redefine what "secure" means, since exploits like Rowhammer and Spectre may make nearly all conventional and cloud computing insecure. Or maybe it's a moving target, like how 64 bit encryption might have once been considered secure but now we need 128 bit or higher.
Edit: the motivation behind this would be to find a container that's 100% secure without emulation, for performance and cost-savings benefits, as well as gaining insights into how to secure operating systems by containerizing their various services.
The only way to make Linux containers a meaningful sandbox is to drastically restrict the syscall API surface available to the sandboxee, which quickly reduces its value. It's no longer a "generic platform that you can throw any workload onto" but instead a bespoke thing that needs to be tuned and reconfigured for every usecase.
This is why you need virtualization. Until we have a properly hardened and memory safe OS, it's the only way. And if we do build such an OS it's unclear to me whether it will be faster than running MicroVMs on a Linux host.
For example there is Kata containers
https://katacontainers.io/
This can be used with regular `podman` by just changing the container runtime so there’s no even need for any extra tooling
In theory you could shove the container runtime into something like k8s
The only meaningful difference is that Linux containers target partitioning Linux kernel services which is a shared-by-default/default-allow environment that was never designed for and has never achieved meaningful security. The number of vulnerabilities resulting from, "whoopsie, we forgot to partition shared service 123" would be hilarious if it were not a complete lapse of security engineering in a product people are convinced is adequate for security-critical applications.
Present a vulnerability assessment demonstrating a team of 10 with 3 years time (~10-30 M$, comparable to many commercially-motivated single-victim attacks these days) can find no vulnerabilities in your deployment or a formal proof of security and correctness otherwise we should stick with the default assumption that software if easily hacked instead of the extraordinary claim that demands extraordinary evidence.
There are VMMs (e.g. pKVM in upstream Linux) with small SLoC that are isolated by silicon support for nested virtualization. This can be found on recent Google Pixel phones/tablets with strong isolation of untrusted Debian Arm Linux "Terminal" VM.
A similar architecture was shipped a decade ago by Bromium and now on millions of HP business laptops, including hypervisor isolation of firmware, "Hypervisor Security : Lessons Learned — Ian Pratt, Bromium — Platform Security Summit 2018", https://www.youtube.com/watch?v=bNVe2y34dnM
Christian Slater, HP cybersecurity ("Wolf") edutainment on nested virt hypervisor in printers, https://www.youtube.com/watch?v=DjMSq3n3Gqs
Seacomp, capabilities, selinux, apparmor, etc.. can help harden containers, but most of the popular containers don't even drop root for services, and I was one of the people who tried to even get Docker/Moby etc.. to let you disable the privileged flag...which they refused to do.
While some CRIs make this easier, any agent that can spin up a container should be considered a super user.
With the docker --privlaged flag I could read the hosts root volume or even install efi bios files just using mknod etc, walking /sys to find the major/minor numbers.
Namespaces are useful in a comprehensive security plan, but as you mentioned, they are not jails.
It is true that both VMs and containers have attack surfaces, but the size of the attack surface on containers is much larger.
Depends I guess as Android has had quite a bit of success with seccomp-bpf & Android-specific flavour of SELinux [0]
> Until we have a properly hardened and memory safe OS ... faster than running MicroVMs on a Linux host.
Andy Tanenbaum might say, Micro Kernels would do just as well.
[0] https://youtu.be/WxbOq8IGEiE
Basically I'd love to see a giant ablation
Edit: when I say anything, I'm not talking user programs. I mean as in, before even the first instruction of the firmware -- before even the virtual disk file is zeroed out, in cases where it needs to be. You literally can't pause the VM during this interval because the window hasn't even popped up yet, and even when it has, you still can't for a while because it literally hasn't started running anything. So the kernel and even firmware initialization slowness are entirely irrelevant to my question.
Why is that?
You also need to start OS services, configure filesystems, prepare caches, configure networking, and so on. If you're not booting UKIs or similar tools, you'll also be loading a bootloader, then loading an initramfs into memory, then loading the main OS and starting the services you actually need, with eachsstep requiring certain daemons and hardware probes to work correctly.
There are tools to fix this problem. Amazon's Firecracker can start a Linux VM in a time similar to that of a container (milliseconds) by basically storing the initialized state of the VM and loading that into memory instead of actually performing a real boot. https://firecracker-microvm.github.io/
On Windows, I think it depends on the hypervisor you use. Hyper V has a pretty slow UEFI environment, its hard disk access always seems rather slow to me, and most Linux distro don't seem to package dedicated minimal kernels for it.
I'm saying it takes a long time for it to even execute a single instruction, in the BIOS itself. Even for the window to pop up, before you can even pause the VM (because it hasn't even started yet). What you're describing comes after all that, which I already understand and am not asking about.
On windows it was almost 10x faster. On the project where this change was released, my morning ritual was to come in, log on, run an svn pull command, lock my screen and go get coffee. I had at least ten minutes to kill after I got coffee, if the pot wasn’t empty when I got there.
Windows is hot garbage about fopen particularly when virus scanning is on.
In practice virtual machines are trying to emulate a lot of stuff that isn’t really needed but they’re doing it for compatibility.
If one builds a hypervisor which is optimized for startup speed and doesn’t need to support generalized legacy software then you can:
> Unlike traditional VMs that might take several seconds to start, Firecracker VMs can boot up in as little as 125ms.
Windows probably has an equivalent.
I'm using Hyper-V and I can connect through XRDP to a GUI Ubuntu 22 in 10 seconds and I can SSH into a Ubuntu 22 server in 3 seconds after start.
For example: your VM starts up with the CPU in 16 bit mode because that’s just how things work in x86 and then it waits for the guest OS to set the CPU into 64 bit mode.
This is completely unnecessary if you just want to run x86-64 code in a virtualized environment and you control the guest kernel and can just assume things are in 64bit mode because it’s not the 70s or whatever
The guest OS would also need to probe few ports to get a bootable disk. If you control the kernel then you can just not do that and boot directly.
There’s a ton of stuff that isn’t needed
A virtual environment doesn’t even really need any BIOS or anything like that.
You can feel free to test with qemu direct kernel booting to see this skips a lot of delay without even having to use a specialized hypervisor like firecracker
If I let a VM use most of my hardware, it takes a few seconds from start to login prompt, which is the same time it takes for my Arch desktop to boot from pressing the button to seeing the login prompt.
That's not what I'm asking.
I'm saying it takes a long time for it to even execute a single instruction, in the BIOS itself. Even for the window to pop up, before you can even pause the VM (because it hasn't even started yet). What you're describing comes after all that, which I already understand and am not asking about.
I think task manager would tell you if there is a blip of memory usage and paging activity at the time. And I'm sure windows itself has profilers that can tell you what is happening when the VM is started..
[0] https://forums.virtualbox.org/viewtopic.php?t=112113
Again, it was a few years ago, but we didn't solve the problem or identify an actual root cause. We stopped banging our heads against that particular wall and switched technologies.
Then the installation instructions include piping a remote script directly to Bash ... Oh irony ...
That said, the concept itself is intriguing.
I'm the creator of microsandbox. If there is anything you need to know about the project, let me know.
This project is meant to make creating microvms from your machine as easy as using Docker containers.
Ask me anything.
The documented code pattern is this:
Due to the way my code works I want to instantiate the sandbox once for a specific class and then have multiple calls to it by class methods, which isn't a clean fit for that "async with" pattern.Any recommendations?
There is an example of that here:
https://github.com/microsandbox/microsandbox/blob/0c13fc27ab...
https://docs.python.org/3/library/contextlib.html#contextlib...
Question: How does networking work? Can I restrict/limit microvms so that they can only access public IP addresses? (or in other words... making sure the microvms can't access any local network IP addresses)
https://github.com/microsandbox/microsandbox/blob/0c13fc27ab...
(also, this project is really cool. great work!)
How is it so fast? Is it making any trade offs vs a traditional VM? Is there potential the VM isolation is compromised?
Can I run a GUI inside of it?
Do you think of this as a new Vagrant?
How do I get data in/out?
It is a lighweight VM and uses the same technology as Firecracker
> Can I run a GUI inside of it?
It is planned but not yet implemented. But it is absolutely possible.
> Do you think of this as a new Vagrant?
I would consider Docker for VMs instead. In a similar way, it focuses on dev ops type use case like deplying apps, etc.
> How do I get data in/out?
There is an SDK and server that help does that and file streaming is planned. But right now, you can execute commands in the VM and get the result back via the server
Native Containers would probably solve here, too.
From https://news.ycombinator.com/item?id=43553198 :
>>> ostree native containers are bootable host images that can also be built and signed with a SLSA provenance attestation; https://coreos.github.io/rpm-ostree/container/
And also from that thread:
> How should a microkernel run (WASI) WASM runtimes?
What is the most minimal microvm for WASM / WASI, and what are the advantages to running WASM workloads with firecracker or microsandbox?
By setting up an image with wasmtime for example.
> and what are the advantages to running WASM workloads with firecracker or microsandbox?
I can think of stronger isolation or when you have legacy stuff you need to run alongside.
> AWS built [Firecracker (which is built on KVM)] to power Lambda and Fargate [2], where they need to quickly spin up isolated environments for running customer code. Companies like E2B use Firecracker to run AI generated code securily in the cloud, while Fly.io uses it to run lightweight container-like VMs at the edge [4, 5].
"We replaced Firecracker with QEMU" (2023) https://news.ycombinator.com/item?id=36666782
"Firecracker's Kernel Support Policy" describes compatible kernel configurations; https://github.com/firecracker-microvm/firecracker/blob/main...
/? wasi microvm kernel [github] https://www.google.com/search?q=wasi+microvm+kernel+GitHub :
- "Mewz: Lightweight Execution Environment for WebAssembly with High Isolation and Portability using Unikernels" (2024) https://arxiv.org/abs/2411.01129 similar: https://scholar.google.com/scholar?q=related:b3657VNcyJ0J:sc...
edit: A fleshed out contributors guide to add support for a new language would help. https://github.com/microsandbox/microsandbox/blob/main/CONTR...
1. each one should have it's own network config, eg so i can use wireguard or a vpn
2. gui pass-through to the host, eg wayland, for trusted tools, eg firefox, zoom or citrix
3. needs to be lightweight. eg gnome-boxes is dead simple to setup and run and it works, but the resource usage was noticeably higher than native
4. optional - more security is better (ie, i might run semi-untrusted software in one of them, eg from a github repo or npm), but i'm not expecting miracles and accept that escape is possible
5. optional - sharing disk with the host via COW would be nice, so i'd only need to install the env-specific packages, not the full OS
i'm currently working on a podman solution, and i believe that it will work (but rebuilding seems to hammer the network - i'm hoping i can tweak the layers to reduce this). does microsandbox offer any advantages for this use case ?
This is possible right now but the networking is not where I want it to be yet. It uses libkrun's default TSI impl; performant and simplifies setup but can be inflexible. I plan to implement an alternative user-space networking stack soon.
> 2. gui pass-through to the host, eg wayland, for trusted tools, eg firefox, zoom or citrix
We don't have GUI passthrough. VNC?
> 3. needs to be lightweight. eg gnome-boxes is dead simple to setup and run and it works, but the resource usage was noticeably higher than native
It is lightweight in the sense that it is not a full vm
> 4. optional - more security is better (ie, i might run semi-untrusted software in one of them, eg from a github repo or npm), but i'm not expecting miracles and accept that escape is possible
The security guarantees are similar to what typical VMs support. It is hardware-virtualized so I would say you should be fine.
> 5. optional - sharing disk with the host via COW would be nice, so i'd only need to install the env-specific packages, not the full OS
Yeah. It uses virtio-fs and has overlayfs on top of that for COW.
Networking continues to be a pain but I'm open to suggestions.
Congratulations on the launch!
That said, hosting microVMs require dedicated hardware or VMs with nested virt support. Containers don’t have that problem.
I wonder how this compares to Orbstack's [0] tech stack on macOS, specifically the "Linux machines" [1] feature. Seems like Orb might reuse a single VM?
---
[0] https://orbstack.dev
[1] https://docs.orbstack.dev/machines/
Firecracker is no different btw and E2B uses that for agentic AI workloads. Anyway, I don't have any major plan except fix some issues with the filesystem rn.
The idea of containers is nice, but they are a hack until kernels are re-architected.
Microsandbox does not offer a cloud solution. It is self-hosted, designed to do what E2B does, to make it easier working with microVM-based sandboxes on your local machine whether that is Linux, macOS or Windows (planned) and to seamlessly transition to prod.
> Do you also use Firecracker under the hood?
It uses libkrun.
I welcome alternatives. It's been tough wrestling with Firecracker and OCI images. Kata container is also tough.
What like about containers is how quickly I can run something, e.g. `docker run --rm ...` without having to specify disk size, amount of cpu cores, etc. I can then diff the state of the container with the image (and other things) to see what some program did while it ran.
So I basically want the same but instead with small vms to have better sandboxing. Sometimes I also use bwrap but it's not really intended to be used on the command line like that.
gVisor has performance problems, though. Their data shows 1/3rd the throughput vs. docker runtime for concurrent network calls--if that's an issue for your use-case.
That is an ideal use case
> Are there better alternatives?
Created microsandbox because I didn't find any
Can we build our own python sandbox using the sandboxfile spec? This is if I want to add my own packages. Would this be just having my own requirements file here - https://github.com/microsandbox/microsandbox/blob/main/MSB_V...
> Can we build our own python sandbox using the sandboxfile spec?
Yes and I plan to make that work with the SDK.
PS: Multi-stage build is WIP.
PS: microsandbox will likely have its own OCI registry in the future
Cloud Hypervisor and Firecracker both have an excellent reputation for ultra lightweight VM's. Both are usable in the very popular Kata Containers project (as well as other upstart VM's Dragonball, & StratoVirt). In us by for example the CNCF Confidential Containers https://github.com/kata-containers/kata-containers/blob/main... https://confidentialcontainers.org/
There's also smaller efforts such as firecracker-containerd or Virtink, both which bring OCI powered microvms into a Docker like position (easy to slot into Kubernetes), via Firecracker and Cloud Hypervisor respectively. https://github.com/smartxworks/virtink https://github.com/firecracker-microvm/firecracker-container...
Poking around under the hood, microsandbox appears to use krun. There is krunvm for OCI support (includes MacOS/arm64 support!). https://github.com/containers/krunvm https://github.com/slp/krun
The orientation as a safe sandbox for AI / MCP tools is a very nicely packaged looking experience, and very well marketred. Congratulations! I'm still not sure why this warrants being it's own project.
However, by looking at it and playing with a few simple examples, I think this is the one that looks the closest so far.
Definitely interested to see the FS support, and also some instruction on how to customize the images to e.g. pre-install common Python packages or Rust crates. As an example, I tried to use the MCP with some very typical use-cases for code-execution that OpenAI/Anthropic models would generate for data analysis, and they almost always include using numpy or a excel library, so you very quicly hit a wall here without the ability to include libraries.
im working on a wrapper that lets you swap runtimes and my first implementation is mostly a wrapper around docker containers
planning to add firecracker next
will explore adding microsandbox too cool stuff!
https://dev.to/rimelek/using-gvisors-container-runtime-in-do...
After this is done, it is:
docker run --rm --runtime=runsc hello-world
So much of the solutions to this stuff I see come from a GitHub repo with a few dozen commits and often a README that says "do not rely on this software yet".
Definitely going to play with it a bit though, I love the idea of hooking into Apple's Hypervisor.framework (which absolutely fits my billion-dollar-company requirement.)
That said I don't think either KataContainer or Cloud Hypervisor has first-class support for macOS.