Interactively Assessing Disentanglement in GANsAnalyzing the Design Space for Visualizing Neural Attention in Text ClassificationTensorsImage tensorsVisualization in Deep LearningInteractive Convolutional Neural NetworkMachine Learning in JS with TensorFlow.js (Part I)How to visualize timeline of a Wiki article?Simulation ResultsRecommender systemsPython in Observable through PyodideArtificial intelligence systems Timelinebackground-position ScrubberDrawings to HumansahajBERT - Bubbles chartPretrained models (image classification)Advanced pretrained models (object detection)Robotics Perturbation: Broken Joint (Rotation)Multi-Agent Evaluation: Object Counting
Machine learning in the browser
slide`# Machine learning in the browser`
slide`# [@ryanseddon](https://twitter.com/ryanseddon)`
slide`# What we'll cover
- High level look at Machine Learning (ML), <br>read [this](https://www.smashingmagazine.com/2019/09/machine-learning-front-end-developers-tensorflowjs/) for a better primer
- What's possible today with demos
`
slide`# What's not covered
- Training your own model
- Transfer learning
- TensorFlow.js API specifics
- No slabs of code`
slide`# Machine Learning has a high barrier to entry`
slide`# Easy to get lost in the lingo
- Linear algebra
- Gradient descent
- Polynomial`
slide.img('https://media3.giphy.com/media/I3fghpZw3o2bu/source.gif', 'https://giphy.com/gifs/so-funny-I3fghpZw3o2bu')
slide`# It's math all the way down`
slide.tweet('1136906133249986562')
slide.img('https://www.rsipvision.com/wp-content/uploads/2015/04/Slide5.png', 'https://www.rsipvision.com/exploring-deep-learning/')
slide.img('https://imgs.xkcd.com/comics/machine_learning.png', 'https://xkcd.com/1838/')
slide`# Machine learning 101
- A process where computers learn by discovering relationships in data
- A model is the description of those relationships
- Inference is applying that model to new data e.g. does <br />this image contain a cat
- Output depends on the problem, probability, label e.g. cat`
slide.tweet('1163058544402411520')
slide`# Don't despair
TensorFlow.js makes it easy to consume their pre-trained models, no PhD in Math required!`
slide`# Let's explore some examples`
slide`# Is the following image a cat?
And what type of cat is it?`
slide.notebook('@ryanseddon/machine-learning-in-js-with-tensorflow-js-part-ii',['input', 'distribution'], {css: 'body{display:flex; justify-content: space-evenly; width: 100%; align-items: center;}'})
slide`# Portrait mode – Bokeh effect`
slide.notebook('@ryanseddon/portrait-mode-using-tensorflow-models-body-pix/2', ['image'])
slide`# Using the BodyPix TensorFlow model
We can get a rough segmentation of a person detected in an image and/or video`
slide.img('https://raw.githubusercontent.com/tensorflow/tfjs-models/master/body-pix/images/body-pix.gif', 'https://github.com/tensorflow/tfjs-models/tree/master/body-pix')
slide.notebook('@ryanseddon/portrait-mode-using-tensorflow-models-body-pix/2', ['image', 'segment'], {css: 'body{display:flex}'})
slide.notebook('@ryanseddon/portrait-mode-using-tensorflow-models-body-pix/2', ['viewof blur','result'])
slide`# Style transfer
Transfer the style of a famous painting onto a photo`
slide`# Only three years ago
- I setup a beefy EC2 instance with a dedicated GPU
- Deep dream not quite style transfer but along the same lines
- Cost me about $15 for 7 minutes work`
slide.tweet('703066890881470464')
slide.img('https://upload.wikimedia.org/wikipedia/commons/thumb/c/c5/Edvard_Munch%2C_1893%2C_The_Scream%2C_oil%2C_tempera_and_pastel_on_cardboard%2C_91_x_73_cm%2C_National_Gallery_of_Norway.jpg/800px-Edvard_Munch%2C_1893%2C_The_Scream%2C_oil%2C_tempera_and_pastel_on_cardboard%2C_91_x_73_cm%2C_National_Gallery_of_Norway.jpg', 'https://en.wikipedia.org/wiki/The_Scream')
slide.notebook('@ryanseddon/style-transfer-with-ml5-js', ['viewof precompute', 'image', 'result2'], { css: 'body{display: grid;}#div0{grid-column-start: 1;grid-column-end: 3; margin-bottom: 20px;}'})
slide`# Why do ML client side?
- Privacy
- Performance/Latency
- Cross platform`
slide`# TensorFlow.js models are large
## For web standards
But it's all in the delivery and how this is presented to the user`
slide.img(await FileAttachment("Screen Shot 2019-10-31 at 2.11.30 pm.png").url(), 'https://storage.googleapis.com/tfjs-models/demos/body-pix/index.html');
slide`# Early days
- The pre-trained model selection is growing
- It's getting easier to train and export to tfjs - see [AutoML](https://medium.com/tensorflow/build-and-deploy-tensorflow-js-models-with-the-power-of-automl-1985b8985083)`
slide`# Where this shines
- PWA app with offline service worker support
- Training a new model, transfer learning
- [Teachable Machine](https://teachablemachine.withgoogle.com/) is a great example
- [Creatability](https://experiments.withgoogle.com/collection/creatability) - Accessible creative tools using ML`
slide.img('https://lh3.googleusercontent.com/ormbIQLOsUJEJdTHlXH-Y7D7fk-42bcpXog6aIhi-r2V-xL5FBpN31KurENIsCOe7EAvHoNS6Eab-r2s0RXemBIaVhJkpw=s850','https://experiments.withgoogle.com/keyboard')
slide`# Open Neural Network Exchange (ONNX)
- Standard format to run most major ML formats
- Everyone in ML but Google
- TensorFlow has the monoply this is their answer
- Less polished browser version ONNX.js`
slide`# WebNN
- Web Neural Network API
- intel proposal to bring ML inference natively to the browser!
- remove the "hack" of WebGL and provide the necessary APIs needed
- [github.com/webmachinelearning/webnn](https://github.com/webmachinelearning/webnn/)`
slide`# Resources
- [tensorflow.org/js](https://www.tensorflow.org/js)
- [ml5.js](https://ml5js.org/) - Makes TensorFlow.js approachable to beginners
- [ONNX.js](https://onnx.ai/) – Alternative to TensorFlow.js
- [This notebook!](fill me in) - observablehq.com is incredible
`
slide`# Thanks`
import { slide, slide_style, attach_events } from '@ryanseddon/slide'
slide_style
attach_events()
One platform to build and deploy the best data apps
Experiment and prototype by building visualizations in live JavaScript notebooks. Collaborate with your team and decide which concepts to build out.
Use Observable Framework to build data apps locally. Use data loaders to build in any language or library, including Python, SQL, and R.
Seamlessly deploy to Observable. Test before you ship, use automatic deploy-on-commit, and ensure your projects are always up-to-date.