Ask a Question

Export data from Dgraph

As an Administrator you can export data on all nodes, configure the Alpha server, specify the export format, export to an object store, disable HTTPS for exports, and encrypt exports

Export data using the GraphQL admin endpoint

You can export the entire data by executing a GraphQL mutation on the /admin endpoint of any Alpha node.

Before you begin:

  • Ensure that there is sufficient space on disk to store the export. Each Dgraph Alpha leader for a group writes output as a gzipped file to the export directory specified through the --export flag (defaults to an export directory). If any of the groups fail because of insufficient space on the disk, the entire export process is considered failed and an error is returned.

  • Make a note of the export directories of the Alpha server nodes. For more information about configuring the Dgraph Alpha server, see Config.

This mutation triggers the export from each of the Alpha leader for a group. Depending on the Dgraph configuration several files are exported. It is recommended that you copy the files from the Alpha server nodes to a safe place when the export is complete.

mutation {
  export(input: {}) {
    response {
      message
      code
    }
  }
}

The export data of the group:

  • in the Alpha instance is stored in the Alpha.
  • in every other group is stored in the Alpha leader of that group.

You need to retrieve the right export files from the Alpha instances in the cluster. Dgraph does not copy all files to the Alpha that initiated the export.

When the export is complete a response similar to this appears:

{"data":{
  "export":{
    "response":{
      "message":"Export completed.",
      "code":"Success"
      }
    }
  },
  "extensions":{
    "tracing":{
      "version":1,
      "startTime":"2022-12-14T07:39:51.061712416Z","endTime":"2022-12-14T07:39:51.129431494Z",
      "duration":67719080
      }
    }
  }

Export data format

By default, Dgraph exports data in RDF format. Replace <FORMAT>with json or rdf in this GraphQL mutation:

mutation {
  export(input: { format: "<FORMAT>" }) {
    response {
      message
      code
    }
  }
}

Export to NFS or a file path

You can override the default folder path by adding the destination input field to the directory where you want to export data. Replace <PATH> in this GraphQL mutation with the absolute path of the directory to export data.

mutation {
  export(input: {
    format: "<FORMAT>"
    destination: "<PATH>"
  }) {
    response {
      message
      code
    }
  }
}

Export to an object store

You can export to an AWS S3, Azure Blob Storage or Google Cloud Storage.

Example mutation to export to AWS S3

mutation {
  export(input: {
    destination: "s3://s3.<region>.amazonaws.com/<bucket-name>"
    accessKey: "<aws-access-key-id>"
    secretKey: "<aws-secret-access-key>"
  }) {
    response {
      message
      code
    }
  }
}
Note The Dgraph URL used for S3 is different than the AWS CLI tools with the aws s3 command, which uses a shortened format: s3://<bucket-name>.

Example mutation to export to MinIO

mutation {
  export(input: {
    destination: "minio://<address>:9000/<bucket-name>"
    accessKey: "<minio-access-key>"
    secretKey: "<minio-secret-key>"
  }) {
    response {
      message
      code
    }
  }
}

Export to a MinIO gateway

You can use MinIO as a gateway to other object stores, such as Azure Blob Storage or Google Cloud Storage.

Azure Blob Storage

You can use Azure Blob Storage through the MinIO Azure Gateway.

Before you begin:

  • Configure a storage account and a Blob container to organize the blobs.
  • Make a note the name of the blob container. It is the <bucket-name> when specifying the destination in the GraphQL mutation.
  • Retrieve storage accounts keys to configure MinIO. Because, MinIO Azure Gateway uses MINIO_ACCESS_KEY and MINIO_SECRET_KEY to correspond to Azure Storage Account AccountName and AccountKey.

You can access Azure Blob Storage locally using one of these methods:

  • Using MinIO Azure Gateway with the MinIO Binary 
    export MINIO_ACCESS_KEY="<AccountName>"
export MINIO_SECRET_KEY="<AccountKey>"
minio gateway azure
  • Using MinIO Azure Gateway with Docker 
    docker run --detach --rm --name gateway \
 --publish 9000:9000 \
 --env MINIO_ACCESS_KEY="<AccountName>" \
 --env MINIO_SECRET_KEY="<AccountKey>" \
 minio/minio gateway azure
  • Using MinIO Azure Gateway with the MinIO Helm chart for Kubernetes: 
    helm repo add minio https://helm.min.io/
helm install my-gateway minio/minio \
  --set accessKey="<AccountName>",secretKey="<AccountKey>" \
  --set azuregateway.enabled=true

You can use the MinIO GraphQL mutation with MinIO configured as a gateway.

Google Cloud Storage

You can use Google Cloud Storage through the MinIO GCS Gateway.

Before you begin:

When you have a credentials.json, you can access GCS locally using one of these methods:

  • Using MinIO GCS Gateway with the MinIO Binary 
    export GOOGLE_APPLICATION_CREDENTIALS="/path/to/credentials.json"
export MINIO_ACCESS_KEY="<minio-access-key>"
export MINIO_SECRET_KEY="<minio-secret-key>"
minio gateway gcs "<project-id>"
  • Using MinIO GCS Gateway with Docker 
    docker run --detach --rm --name gateway \
  --publish 9000:9000  \
  --volume "</path/to/credentials.json>":/credentials.json \
  --env GOOGLE_APPLICATION_CREDENTIALS=/credentials.json \
  --env MINIO_ACCESS_KEY="<minio-access-key>" \
  --env MINIO_SECRET_KEY="<minio-secret-key>" \
  minio/minio gateway gcs "<project-id>"
  • Using MinIO GCS Gateway with the MinIO Helm chart for Kubernetes: 
    ## create MinIO Helm config
cat <<-EOF > myvalues.yaml
accessKey: <minio-access-key>
secretKey: <minio-secret-key>

gcsgateway:
  enabled: true
  projectId: <project-id>
  gcsKeyJson: |
$(IFS='\n'; while read -r LINE; do printf '    %s\n' "$LINE"; done < "</path/to/credentials.json>")
EOF

## deploy MinIO GCS Gateway
helm repo add minio https://helm.min.io/
helm install my-gateway minio/minio \
  --values myvalues.yaml

You can use the MinIO GraphQL mutation with MinIO configured as a gateway.

Disable HTTPS for exports to S3 and Minio

By default, Dgraph assumes the destination bucket is using HTTPS. If that is not the case, the export fails. To export to a bucket using HTTP (insecure), set the query parameter secure=false with the destination endpoint in the destination field:

mutation {
  export(input: {
    destination: "minio://<address>:9000/<bucket-name>?secure=false"
    accessKey: "<minio-access-key>"
    secretKey: "<minio-secret-key>"
  }) {
    response {
      message
      code
    }
  }
}

Use anonymous credentials

When exporting to S3 or MinIO where credentials are not required, can set anonymous to true.

mutation {
  export(input: {
    destination: "s3://s3.<region>.amazonaws.com/<bucket-name>"
    anonymous: true
  }) {
    response {
      message
      code
    }
  }
}

Encrypt exports

Export is available wherever an Alpha is running. To encrypt an export, the Alpha must be configured with the --encryption key-file=value.

Note The --encryption key-file was used for Encryption at Rest and will now also be used for encrypted exports.

Use curl to trigger an export

This is an example of how you can use curl to trigger an export.

  1. Create GraphQL file for the desired mutation: 
    cat <<-EOF > export.graphql
mutation {
  export(input: {
    destination: "s3://s3.<region>.amazonaws.com/<bucket-name>"
    accessKey: "<aws-access-key-id>"
    secretKey: "<aws-secret-access-key>"
  }) {
    response {
      message
      code
    }
  }
}
EOF
  2. Trigger an export with curl 
    curl http://localhost:8080/admin --silent --request POST \
  --header "Content-Type: application/graphql" \
  --upload-file export.graphql
Ask a Question Edit the page Report an issue