> ## Documentation Index
> Fetch the complete documentation index at: https://lightdash-mintlify-20de881e.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure Lightdash to use external storage

> Set up the required S3-compatible bucket on GCP, AWS, Azure, or MinIO

On Lightdash, we generate some files like:

* Images for Slack unfurl and scheduled deliveries
* Results on JSONL/CSV/Excel format

These files need to be stored in a S3 Compatible Cloud storage. Some options are GCP Buckets, S3 Storage and MinIO.

<Warning>
  S3-compatible storage is **required**. Lightdash will fail to start if `S3_ENDPOINT`, `S3_BUCKET`, and `S3_REGION` are not set. All warehouses (including Snowflake) fetch async query results from S3.
</Warning>

## Configure cloud storage on Google Cloud Platform

In the Google Cloud console, go to Cloud Storage and create a bucket with these details:

* Give it a unique `Bucket name`, such as `lightdash-cloud-file-storage-eu`
* Select a region, such as multi-region US, multi-region EU, or a single region like `europe-west1`
* Select the default `Standard` storage class
* Enable `Enforce public access prevention on this bucket`
* Select `Uniform` access control
* Set Protection to `none`

Set `S3_ENDPOINT` to `https://storage.googleapis.com` and `S3_REGION` to `auto`.

Then choose how Lightdash authenticates to the bucket. Google Cloud Storage accepts two
methods, and they are not interchangeable.

### Option A: HMAC keys

Use this option anywhere outside Google Kubernetes Engine (GKE), or when you want the
simplest setup.

1. Go to **Settings > Interoperability** and create an **Access key for service account**.
2. Copy the access key and the secret.
3. Set `S3_ACCESS_KEY` and `S3_SECRET_KEY`.

These keys are long-lived static secrets. Rotate them on a schedule. Google Cloud Storage
supports several active keys per service account, so you can rotate without downtime.

### Option B: Workload identity, with no keys

Use this option when Lightdash runs on GKE and you do not want static secrets in the
cluster. Lightdash sends a short-lived Google OAuth token instead of signing requests with
a key.

Set `S3_AUTH_MODE=gcp_oauth` and leave `S3_ACCESS_KEY` and `S3_SECRET_KEY` unset.

<Warning>
  `S3_USE_CREDENTIALS_FROM` does not enable this option, even though its `ecs` and `ec2`
  values look like they should. Every source that setting accepts is an **AWS** credential
  provider, and Google Cloud Storage cannot verify AWS credentials. On GKE, use
  `S3_AUTH_MODE=gcp_oauth`.
</Warning>

Grant these permissions to the Google service account that the pod runs as:

1. Bind the Kubernetes service account to the Google service account, as described in the
   [GKE workload identity documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity).
2. Grant the `roles/storage.objectUser` role **on the bucket**. Do not grant a project-wide
   role such as Editor.
3. Grant the `roles/iam.serviceAccountTokenCreator` role **on the service account itself**.
   The member and the resource are the same service account. Lightdash needs this role to
   sign download URLs.
4. Enable the `iamcredentials.googleapis.com` API in the project.

<Warning>
  If you skip step 3, the deployment looks healthy. Uploads, downloads, and result caching
  all work. Only signed download links fail, and only when a user asks for one. The error is
  `Permission 'iam.serviceAccounts.signBlob' denied on resource`.
</Warning>

Set these environment variables:

```bash theme={null}
S3_ENDPOINT=https://storage.googleapis.com
S3_REGION=auto
S3_BUCKET=your-bucket-name
S3_AUTH_MODE=gcp_oauth
S3_FORCE_PATH_STYLE=true
```

Both URL styles work with Google Cloud Storage. Path style also works for bucket names that
contain dots, which break TLS certificate matching in the other style.

Signed URLs in this mode last at most 7 days, which is a Google Cloud Storage limit.
Lightdash fails to start if `S3_EXPIRATION_TIME` exceeds 604800 seconds.

To develop locally against a bucket that uses this mode, run:

```bash theme={null}
gcloud auth application-default login --impersonate-service-account=SERVICE_ACCOUNT_EMAIL
```

Your own user credentials cannot sign URLs, and the error is
`Cannot sign data without 'client_email'`. Impersonating the service account fixes it. Do
not download a service account key file.

## Configure cloud storage on AWS

* Navigate to the S3 section of the AWS Management Console and click on the Create Bucket button.
* Give your bucket a name and select the region where you want to store your data.
* Next, you need to set the permissions for your bucket. Make it private.

To export your S3 credentials, you need to follow these steps:

* Navigate to the IAM section and click on the Users tab.
* Click on the user whose credentials you want to export.
* Click on the Security Credentials tab and locate the Access Keys section.
* Click on the Create Access Key button.
* Download the CSV file that contains your Access Key ID and Secret Access Key.

Check [this guide](https://docs.aws.amazon.com/general/latest/gr/s3.html) to see what's the right `S3_ENDPOINT` for your bucket

## Configure cloud storage using MinIO

Creating a bucket in MinIO

* Login to the MinIO console and click on "Buckets" in the side bar
* Click on "Create Bucket"
* Give your bucket a name and click "Create Bucket"

Creating access credentials in MinIO

* Click on "Access Keys" in the side bar
* Click "Create access key"
* Give a name to your new access key and click "Create"
* Download the JSON file containing both your Access Key ID and Secret Access Key

MinIO needs path style bucket URLs, for this you will need to set `S3_FORCE_PATH_STYLE: true` in your environment variables.

## Azure Storage

Azure Blob Storage is not natively compatible with the S3 API. While Lightdash supports external object storage by allowing integration with S3-compatible APIs, Azure's storage service does not provide this compatibility out of the box. This means that you **cannot use Azure Blob Storage as a drop-in replacement for S3** in Lightdash deployments.

Instead, you can use one of the following S3-compatible solutions within your Azure setup:

* [**MinIO**](https://min.io)**:** S3-compatible object storage
* [**s3proxy**](https://github.com/gaul/s3proxy)**:** A lightweight proxy that adds an S3-compatible API layer on top of Azure Blob Storage.

## Configure Lightdash to use S3 credentials

To enable Lightdash to use your S3 bucket for cloud storage, you'll need to set the following environment variables:

```bash theme={null}
S3_ENDPOINT        # required
S3_REGION          # required
S3_BUCKET          # required
S3_ACCESS_KEY      # optional if using IAM role
S3_SECRET_KEY      # optional if using IAM role
S3_EXPIRATION_TIME # optional, defaults to 259200 seconds (3 days)
```

For a comprehensive list of all possible S3-related environment variables and other configurations, please visit the [Environment Variables documentation](/self-host/customize-deployment/environment-variables#s3).

### Use separate storage for query results

Query results use the main S3 configuration by default. To store them with a different S3-compatible provider, set the `RESULTS_S3_*` variables that differ from it — credentials and path-style addressing have `RESULTS_S3_*` equivalents too:

```bash theme={null}
RESULTS_S3_ENDPOINT=https://s3.eu-west-1.amazonaws.com
RESULTS_S3_REGION=eu-west-1
RESULTS_S3_BUCKET=my-lightdash-query-results
```

Any omitted result storage setting inherits its corresponding `S3_*` value.

<Warning>
  Raw JSONL result downloads use a presigned URL that browsers fetch directly from `RESULTS_S3_ENDPOINT`. This endpoint must be reachable from users' browsers; `S3_PUBLIC_ENDPOINT` does not apply to the query results bucket. The bucket can remain private, but an IP policy that limits `GetObject` to Lightdash's backend egress addresses will block these downloads.
</Warning>

### Authenticate without static keys

How you do this depends on the cloud provider, because the two mechanisms are unrelated.

**On AWS.** Omit `S3_ACCESS_KEY` and `S3_SECRET_KEY`, and the AWS SDK looks for an IAM role
automatically. To control which credential sources it tries and in what order, set
`S3_USE_CREDENTIALS_FROM`. For more detail, read the
[AWS SDK for JavaScript documentation on setting credentials in Node.js](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-credentials-node.html).

**On Google Cloud.** Set `S3_AUTH_MODE=gcp_oauth`. See
[Option B: workload identity, with no keys](#option-b-workload-identity-with-no-keys).
`S3_USE_CREDENTIALS_FROM` does not work here, because every source it accepts is an AWS
credential provider.

Signed URLs last at most 7 days in both cases, whatever you set `S3_EXPIRATION_TIME` to.
The limit comes from the signing algorithm, not from Lightdash.

## Production bucket strategy

For an enterprise deployment, use a **dedicated bucket per purpose**, each with a scoped credential that can only touch its own bucket. This is how Lightdash Cloud sets up storage:

* **Results / cache bucket** — add a lifecycle rule deleting objects after **1 day**. Results are ephemeral, so this keeps the bucket small and cheap.
* **Pre-aggregations bucket** (if using [pre-aggregates](/self-host/nats-workers/pre-aggregate-workers)) — lifecycle delete after **\~30 days**.
* **Data-apps bucket** (if using [data apps](/self-host/enterprise-features/data-apps)) — persistent, with **no** delete lifecycle.

**Block all public access** on every bucket, and prefer short-lived credentials over static
keys. On EKS, use IAM roles for service accounts (IRSA). On GKE, set
`S3_AUTH_MODE=gcp_oauth` to use workload identity. In both cases, signed URLs last at most
7 days.

If users share download links that must outlive the signed-URL TTL (`S3_EXPIRATION_TIME`, default 3 days), enable `PERSISTENT_DOWNLOAD_URLS_ENABLED: "true"` — see [persistent download URLs](/self-host/customize-deployment/environment-variables#persistent-download-urls).