Ian Johnson

Observable

Templates: Integrations

Observable’s first-party templates, collected here, are displayed on https://observablehq.com/templates

Templates

It may be helpful to link to the 'Secrets' documentation: https://observablehq.com/@observablehq/secrets
An important caveat of using secrets to store credentials is that they work only for private (non-published) personal notebooks or for private Teams notebooks (as shared secrets).  Once published they will break.  Thus, if one intends to use the AWS SDK for JS to provide a dashboard view of S3-hosted data to share out to others, one would need to supply visitors to the site with an alternative means of storing credentials.  One (not entirely secure but workable) approach is to store keys in local memory.

Aaron Kyle Dennis

Agreed that the usage of Secrets makes this targeted at internal team and private use cases. The Secrets page linked does have further documentation on it.

This notebook is currently broken: "Error: [Package Error] "@aws-sdk/util-utf8-browser" no dependency version info found. (Imported by "@aws-crypto/sha256-browser")."

See also:
- https://github.com/aws/aws-sdk-js-v3/issues/2810
- https://github.com/aws/aws-sdk-js-v3/issues/2307
- https://talk.observablehq.com/t/templates-integrations-aws-s3-api-starter-error-with-aws-sdk-client-s3-import/5744

Fabian Iwand

FYI, the dependency was added just a few hours ago to sha256-browser (https://github.com/aws/aws-sdk-js-crypto-helpers/pull/245) and a new npm release has been published (https://cdn.jsdelivr.net/npm/@aws-crypto/sha256-browser/package.json).

This seems to have been fixed in client-s3@v3.39.0.

Toph Tucker

"RuntimeError: [Package Error] "@aws-sdk/querystring-parser@v3.23.0" could not be built. (Imported by "@aws-sdk/url-parser")."

Tom Larkworthy

thanks for pointing it out, fixed with version pin to v3.22.0

This notebook glosses over this process quite a bit... It's important to note that the table above is only visible from the Templates page because the bucket contents are publicly exposed (which can be verified by directly looking at the S3 bucket: https://pinyin.s3.amazonaws.com/). If you have more restrictive view permissions, this table would not be visible to non-credentialed users, and hence would return an error on a public notebook such as this one.  For any viewer to navigate to this page and not see an error, one first has to set the AWS S3 bucket policies.  In other words, there's quite a bit of back-end AWS configuration work involved in setting up a working AWS connection from Observable that isn't elaborated here.

We actually use a file attachment to show demo data on the public template, and the fact that the bucket is configured to be public does enable individual items to be viewed. This is intended to be used in a secure way (private or team shared notebook), which will certainly necessitate some setup in AWS. We imagine a more common use case would be the bucket is configured to only be visible to the IAM user who's API keys are stored in the Secrets. This way the API could serve the directories and files and everything is loaded securely.

Hi Ian, I'm seeing a similar error to what Tom originally pointed out in July: 

RuntimeError: [Package Error] "@aws-sdk/util-utf8-browser" no dependency version info found. (Imported by "@aws-crypto/sha256-browser").

It looks like something broke in a recent AWS release? All of my notebooks which relied on v3.31.0 are also broken. Any ideas you have for how to fix this would be much appreciated.

Sara Quigley

import("https://cdn.skypack.dev/pin/@aws-sdk/client-s3@v3.22.0-BOVgipyZxgVrYP0YhWWo/mode=imports/optimized/@aws-sdk/client-s3.js") seems to work (this is the underlying pinned URL from skypack if you look at https://cdn.skypack.dev/@aws-sdk/client-s3@v3.22.0).

Visnu Pitiyanuvath

er, maybe I mean https://cdn.skypack.dev/pin/@aws-sdk/client-s3@v3.22.0-BOVgipyZxgVrYP0YhWWo/mode=imports,min/optimized/@aws-sdk/client-s3.js

Weirdly, the above fix only works for me in Safari, not Chrome or Firefox. Looks like the issue is being tracked here: https://github.com/aws/aws-sdk-js-v3/issues/2810. I'll keep looking…

Thanks for your help, Visnu and Toph. I really appreciate it.

HI there, the fix that Visnu mentioned doesn't seem to consistently work in Safari, are there any other workarounds that you can recommend? Thank you.

What about using this for the require? `require('aws-sdk@2.376.0/dist/aws-sdk-react-native.js')`

sorry, old version. try: AWS = require('aws-sdk@2.1009.0/dist/aws-sdk.min.js')

Thanks Aaron. Using the require works, but I am then trying to read the stream of objects using the GetObjectCommand, as in this example: https://observablehq.com/d/931e9dc88f86cc80

Do you know what else I need to change if in order to not get an error when invoking AWS.GetObjectCommand? 

Thank you very much

Is your user appropriately credentialed? As is, you've defined both `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`?

Yes. Everything worked fine until last Wednesday 10/13

I took a different approach in this notebook using the old SDK.js (the first one I suggested - react native). It generally works: https://observablehq.com/d/f6eb7116e8e7be30

Aaron, thank you so much. Your different approach works for me. I am so grateful for your help. Best regards, Sara

This seems to have been fixed in AWS’s client-s3@v3.39.0. So sorry for the breakage; it seems to still be early days for v3 of the SDK, hopefully it'll settle down.

Thanks Toph! It's certainly not your fault, as the AWS s3 client is beyond Observable's control. I do appreciate all of the assistance with troubleshooting though. :)