slide`# Machine learning in the browser`
slide`# [@ryanseddon](https://twitter.com/ryanseddon)`
slide`# What we'll cover
- High level look at Machine Learning (ML), <br>read [this](https://www.smashingmagazine.com/2019/09/machine-learning-front-end-developers-tensorflowjs/) for a better primer
- What's possible today with demos
`
slide`# What's not covered
- Training your own model
- Transfer learning
- TensorFlow.js API specifics
- No slabs of code`
slide`# Machine Learning has a high barrier to entry`
slide`# Easy to get lost in the lingo
- Linear algebra
- Gradient descent
- Polynomial`
slide.img('https://media3.giphy.com/media/I3fghpZw3o2bu/source.gif', 'https://giphy.com/gifs/so-funny-I3fghpZw3o2bu')
slide`# It's math all the way down`
slide.tweet('1136906133249986562')
slide.img('https://www.rsipvision.com/wp-content/uploads/2015/04/Slide5.png', 'https://www.rsipvision.com/exploring-deep-learning/')
slide.img('https://imgs.xkcd.com/comics/machine_learning.png', 'https://xkcd.com/1838/')
slide`# Machine learning 101
- A process where computers learn by discovering relationships in data
- A model is the description of those relationships
- Inference is applying that model to new data e.g. does <br />this image contain a cat
- Output depends on the problem, probability, label e.g. cat`
slide.tweet('1163058544402411520')
slide`# Don't despair
TensorFlow.js makes it easy to consume their pre-trained models, no PhD in Math required!`
slide`# Let's explore some examples`
slide`# Is the following image a cat?
And what type of cat is it?`
slide.notebook('@ryanseddon/machine-learning-in-js-with-tensorflow-js-part-ii',['input', 'distribution'], {css: 'body{display:flex; justify-content: space-evenly; width: 100%; align-items: center;}'})
slide`# Portrait mode – Bokeh effect`
slide.notebook('@ryanseddon/portrait-mode-using-tensorflow-models-body-pix/2', ['image'])
slide`# Using the BodyPix TensorFlow model
We can get a rough segmentation of a person detected in an image and/or video`
slide.img('https://raw.githubusercontent.com/tensorflow/tfjs-models/master/body-pix/images/body-pix.gif', 'https://github.com/tensorflow/tfjs-models/tree/master/body-pix')
slide.notebook('@ryanseddon/portrait-mode-using-tensorflow-models-body-pix/2', ['image', 'segment'], {css: 'body{display:flex}'})
slide.notebook('@ryanseddon/portrait-mode-using-tensorflow-models-body-pix/2', ['viewof blur','result'])
slide`# Style transfer
Transfer the style of a famous painting onto a photo`
slide`# Only three years ago
- I setup a beefy EC2 instance with a dedicated GPU
- Deep dream not quite style transfer but along the same lines
- Cost me about $15 for 7 minutes work`
slide.tweet('703066890881470464')
slide.img('https://upload.wikimedia.org/wikipedia/commons/thumb/c/c5/Edvard_Munch%2C_1893%2C_The_Scream%2C_oil%2C_tempera_and_pastel_on_cardboard%2C_91_x_73_cm%2C_National_Gallery_of_Norway.jpg/800px-Edvard_Munch%2C_1893%2C_The_Scream%2C_oil%2C_tempera_and_pastel_on_cardboard%2C_91_x_73_cm%2C_National_Gallery_of_Norway.jpg', 'https://en.wikipedia.org/wiki/The_Scream')
slide.notebook('@ryanseddon/style-transfer-with-ml5-js', ['viewof precompute', 'image', 'result2'], { css: 'body{display: grid;}#div0{grid-column-start: 1;grid-column-end: 3; margin-bottom: 20px;}'})
slide`# Why do ML client side?
- Privacy
- Performance/Latency
- Cross platform`
slide`# TensorFlow.js models are large
## For web standards
But it's all in the delivery and how this is presented to the user`
slide.img(await FileAttachment("Screen Shot 2019-10-31 at 2.11.30 pm.png").url(), 'https://storage.googleapis.com/tfjs-models/demos/body-pix/index.html');
slide`# Early days
- The pre-trained model selection is growing
- It's getting easier to train and export to tfjs - see [AutoML](https://medium.com/tensorflow/build-and-deploy-tensorflow-js-models-with-the-power-of-automl-1985b8985083)`
slide`# Where this shines
- PWA app with offline service worker support
- Training a new model, transfer learning
- [Teachable Machine](https://teachablemachine.withgoogle.com/) is a great example
- [Creatability](https://experiments.withgoogle.com/collection/creatability) - Accessible creative tools using ML`
slide.img('https://lh3.googleusercontent.com/ormbIQLOsUJEJdTHlXH-Y7D7fk-42bcpXog6aIhi-r2V-xL5FBpN31KurENIsCOe7EAvHoNS6Eab-r2s0RXemBIaVhJkpw=s850','https://experiments.withgoogle.com/keyboard')
slide`# Open Neural Network Exchange (ONNX)
- Standard format to run most major ML formats
- Everyone in ML but Google
- TensorFlow has the monoply this is their answer
- Less polished browser version ONNX.js`
slide`# WebNN
- Web Neural Network API
- intel proposal to bring ML inference natively to the browser!
- remove the "hack" of WebGL and provide the necessary APIs needed
- [github.com/webmachinelearning/webnn](https://github.com/webmachinelearning/webnn/)`
slide`# Resources
- [tensorflow.org/js](https://www.tensorflow.org/js)
- [ml5.js](https://ml5js.org/) - Makes TensorFlow.js approachable to beginners
- [ONNX.js](https://onnx.ai/) – Alternative to TensorFlow.js
- [This notebook!](fill me in) - observablehq.com is incredible
`
slide`# Thanks`
import { slide, slide_style, attach_events } from '@ryanseddon/slide'
slide_style
attach_events()
Purpose-built for displays of data
Observable is your go-to platform for exploring data and creating expressive data visualizations. Use reactive JavaScript notebooks for prototyping and a collaborative canvas for visual data exploration and dashboard creation.
