Generating non-expiring signed URLs on AWS
Amazon S3 has a maximum signature expiry of 7 days. To upload signed URLs that don’t expire, we recommend proxying URLs through an endpoint on your server.
This example demonstrates how to proxy through a simple flask app. We provided a one-click deploy through Heroku, but you can also build this simple handler into your existing web service.
Step 1: Deploy a proxy endpoint
This endpoint accepts a signed URL with our JWT secret and returns a new, signed s3 URL to an asset.
First, you’ll need to get IAM information to be able to create pre-signed URLs:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
- a bucket name
Make sure that this IAM user can LIST and GET files in the bucket.
When your proxy is ready, you can deploy it with one click here:
Step 2: Generate signed URLs pointing at our proxy
For each asset in our s3 bucket, we will generate a signed URL with our JWT secret that points to our server endpoint.
-
From Heroku, click open app to get the host URL of your new app. Then, get the generated secret (
settings
>reveal config vars
). -
Clone the
generate-tokenized-urls
repository:
git clone https://github.com/kili-technology/generate-tokenized-urls
cd generate-tokenized-urls/
- Create your URLs:
// confirm you have node.js installed
node --version
npm install
node cli.js
--bucket <your-aws-bucket-name>
--host https://<your-new-heroku-url>.herokuapp.com/
--secret <heroku-generated-config-secret>
--output kili-import.json
This code creates the kili-import.json
file. Import the URLs from kili-import.json
to Kili and you’re good to go.
Updated 9 months ago