Magic .env files built for sharing: Human-first, AI-friendly

This is about formalizing required secrets for dev. Many repos have a ".env.sample", which is better than nothing - you copy it to .env and fill in the blanks. If this is done automatically, even better. If the automatic .env setup also loads secrets from your password manager (with prompting), even better.

    export OPENAPI_API_KEY=$(op read "op://api-prod/openai/api-key")

It's such a shame this isn't built into shells. There isn't really a security issue here as you have to trust the directory first before it fires.

I agree that loading environment variables should not be the responsibility of the process which consumes them, not sure what the benefit of renaming .env to .envrc is.

I use mise to load environment variables from .env, since I also use it to manage tool versions. When I don't have that available I just do

  set -a; source .env; set +a

direnv is definitely another good option.

And I don't get storing your configuration in env variables, period. They're not discoverable at all, there's no namespacing nor any other isolation, there's no structure to them, and if you want a list you'll have to come up with an ad-hoc encoding. Environment variables are a mess, you'd be better off with a config file.

Most tech people don't grok the abstraction between an executable program and the environment it is executed in.

The thing that runs your code is responsible for adding the environment variables from wherever they are kept for that environment. The execution environments running your code are not checking out your Git repository and reading files from it before they execute your program; that would introduce a chicken-and-egg problem (and make it much harder to run your program in new environments).

Below is an example of why you can't just load all your variables in a single ".env" file.

Environments:

    - *development*:
        host: "my laptop"
        variables: [FOO=dev]
  
    - *staging*:
        host: "a ci/cd server"
        variables: [FOO=stage]

    - *production*:
        host: "a kubernetes server"
        variables: [FOO=prod]

I want to run my program in development!

    ->> open a terminal on my MacOS laptop
             ->> change to directory with my code
  
             ->> load environment variables into shell
                 $ source .env
  
             ->> run my program
                 $ ./MyProgram.EXE
             ->> the shell takes the variables it has, and adds them to the program at runtime
                 char *const argv[] = {"/Users/me/app_src/MyProgram.EXE", NULL};
                 char *const envp[] = {"FOO=dev", NULL};
                 execve("/Users/me/app_src/MyProgram.EXE", argv, envp);
  
                 ->> the program is loaded, executed, and accesses environment variables passed to it

I want to run my program in staging!

    ->> start a job on a ci/cd server
         ->> the ci/cd server starts a Docker container
           ->> the ci/cd server begins running commands in the Docker container
               ->> checkout latest code and change to new directory
  
               ->> load environment variables into shell
                   ->> retrieves environment variables set in ci/cd job configuration
                   ->> sets those variables in the shell
  
               ->> run my program
                   $ ./MyProgram.EXE
               ->> the shell takes the variables it has, and adds them to the program at runtime
                   char *const argv[] = {"/Users/me/app_src/MyProgram.EXE", NULL};
                   char *const envp[] = {"FOO=stage", NULL};
                   execve("/Users/me/app_src/MyProgram.EXE", argv, envp);
  
                   ->> the program is loaded, executed, and accesses environment variables passed to it

I want to run my program in production!

    ->> start a job on a kubernetes server
         ->> a deployment scheduler schedules a pod to start\
  
             ->> it looks up a configmap with environment variables
             ->> it looks up a secrets object with secrets
             ->> it sets environment variables in the pod based on the configmaps and secrets
  
             ->> it starts a container in the pod
               ->> it begins running commands in the container
                   ->> run my program
                       $ ./MyProgram.EXE
                   ->> the container takes the variables it has, and adds them to the program at runtime
                       char *const argv[] = {"/Users/me/app_src/MyProgram.EXE", NULL};
                       char *const envp[] = {"FOO=prod", NULL};
                       execve("/Users/me/app_src/MyProgram.EXE", argv, envp);
  
                       ->> the program is loaded, executed, and accesses environment variables passed to it

You have to add the environment variables to the execution environment before your program ever gets run. Otherwise (for example) you could never pull a Git repository to load variables, because where would the credentials to pull your Git repository come from? They have to be added beforehand. So that beforehand step is where you add all your environment variables for that environment. Your program merely reads them when it is executed; no need for your program to read additional files.

You should not do something like keep a ".env.dev", ".env.stage", ".env.prod", packaged up in a container. Each execution environment may need slightly different settings at run time. And secrets should not be kept in your Git repo, they should be loaded as-needed, at runtime, by your execution environment.

All this is covered neatly in The Twelve Factor App (https://12factor.net/config). As someone who's been doing this for two decades, I highly recommend everyone follow their guide to the letter.

Tons of folks have plaintext secrets in their env files which are leaked via AI assistants every day. By getting those out of plaintext, the risk is totally removed.

The schema itself (and the automatic types it can generate) also gives AI more context about what configuration is available, and what each item is for.

Buzzwords!

We're still working on the copy. Any specific feedback is appreciated.

Note that you can load env vars into anything via varlock run, not just javascript. The JS integration is a bit deeper, providing automatic type generation, log redaction, and leak prevention.

Right now, you can use any provider that has a CLI to fetch a single secret. In the near future, we'll be adding native plugins to make these integrations even easier.

Vault can be a huge lift and doesn't make sense for many projects - we wanted to build a tool that makes sense from day one, even when there is no backing provider, but can grow with your team and change providers seamlessly.

On most projects you end up wiring up a bunch of custom logic to handle your config, which is often injected as environment variables - think loading from a secure source, validation logic, type safety, pre-commit git scanning, etc.

It's annoying to do it right, so people often take shortcuts - skip adding validation, send files over slack, don't add docs, etc...

The common pattern of using a .env.example file leads to constant syncing problems, and we often have many sources of truth about our config (.env.example, hand-written types, validation code, comments scattered throughout the codebase)

This tool lets you express additional schema info about the config your application needs via decorators in a .env file, and optionally set values, either directly if they are not sensitive, or via calls to an external service. This shouldn't be something we need to recreate when scaffolding out every new project. There should be a single source of truth - and it should work with any framework/language.

This is exactly the kind of thing we are trying to replace. No more hacky custom solutions needed. With varlock you get type safety, validation, and the ability to compose together your config however you need to, without having to reinvent the wheel.

its an old library but convict is great for this in the nodejs ecoystem

We say "made for sharing" because .env.schema replaces .env.example which always drifts from reality - and often requires insecurely sharing secrets manually.

Even if not setting values within your files, you can rely entirely on env vars in the platform where the code runs and still benefit from validation provided by varlock.

Right now we give 1Password as an example, but you can use any provider that has a CLI. We are also working on a plugin system that should make it easier to integrate with any provider.

As for redaction - that note is about how we redact your secrets from _our_ CLI output. However we also provide tools to redact within your application. Right now this works only in JavaScript, by patching global console methods. We will also hook into stdout for varlock run, similar to what the 1Password cli does.

The leak detection is much more interesting - especially in hybrid client/server frameworks, where you can easily shoot yourself in the foot.

By removing plaintext secrets from env files, we totally remove the risk of them leaking via AI code assistants, which I guarantee is happening millions of times a day right now. Also the schema itself and autogenerated types give AI much more context about your env.

I don’t know about varlock, but 1Password’s `op` CLI tool seems to hook the STDOUT pipe and find/replace any instances of the secrets with “concealed by 1Password”. It works even if I drop into a REPL and try every way I can think of to print it out to the console.

So it would seem, on that front, that 1Password is doing the heavy lifting.

Using 1Password in this way has proven way better than storing .env files in plain text on dev machines, where the .env files get picked up if a company does backups, or someone stores a repo in their Dropbox folder, file gets flagged as potential malware and uploaded somewhere for further analysis, etc.