2d:null 2e:["$","title",null,{"children":"\"Learning To See\" - ML Project Presentation | ychsu6 | Observable"}] 33:T53d,## Memo Akten [Project Website](https://www.memo.tv/works/learning-to-see/)
[PHD Thesis](https://research.gold.ac.uk/id/eprint/30191/1/COM_thesis_AktenM_2021.pdf)
[Paper presented at SIGGRAPH 2019](https://dl.acm.org/doi/10.1145/3306211.3320143) ### Project Brief - Interactive installation in which a number of neural networks analyse a live camera feed pointing at a table with everyday objects. - Audience can manipulate the objects on the table and see corresponding scenery emerging on the display, in real time, reinterpreted by the neural networks. - Scene changes between different networks trained on five different datasets(four natural elements): Ocean & waves (representing ‘water’); Clouds & Sky (representing ‘air’); Fire & Flowers (representing earth, and life); Hubble Space Telescope (representing the universe, cosmos, quintessence, aether, the void, the home of God).

34:T4c9,#### Training pix2pix, a conditional GAN, designed for paired image-to-image translation. The generator network learns a mapping G : {input, z} 7→ target, where z is a random noise vector within the latent distribution of the model, input is an input image, and target is the desired target image; therefore, training data for pix2pix models typically require (input, target) image pairs. Only explicitly provide target images and do not provide input images. Constructs input images from the target images during training.
Training and inference stages. During pix2pix training, the images used as inputs in the (input, target) pairs, are generated from the target images via a custom image processing pipeline. The parameters of this pipeline are randomized on every iteration to aid generalisation. During inference, the video feed is also fed through the same image processing pipeline. In addition, a number of additional image processing filters are applied before the images are fed into the model for prediction.

35:T657,## [Learning to See: Hello, World! (2017)](https://www.memo.tv/works/learning-to-see-hello-world/) - Artificial Neural Network starts blank, and trains live on a camera feed. - A Deep Neural Network (DNN) opens its eyes for the first time and tries to "understand" what it sees as it trains in real-time on a live camera feed. - Users manipulate a number of hyperparameters (learning rate, momentum, gradient clipping thresholds, optimizer function, loss function...etc.) in real-time (e.g. using faders on a midi controller). - Real-time results help build a qualitative understanding of how such hyperparameters impact the training process. Show how DNN learns. - This neural network has not been trained on anything. It starts off completely blank (i.e. randomly initialized). "Understanding’ means trying to find patterns, trying to find regularities in what it’s seeing, so that it can efficiently compress and organize incoming information in context of its past experience and make accurate and efficient predictions of the future. - Unexpected Behaviours: the network also ‘forgets’. When too much new information comes in, and it doesn’t re-encounter past experiences, it slowly loses those filters and representations required to reconstruct those past experiences.
36:[{"id":"47ff7cd1e859c73f05c00d4de7f8b02448123c6ad636a8a325ab3971add3811abf1ba8277e0d03e6f2f6ab171160e355da74bf7b35737552afd4ee21f4c3afc0","href":"https://static.observableusercontent.com/files/47ff7cd1e859c73f05c00d4de7f8b02448123c6ad636a8a325ab3971add3811abf1ba8277e0d03e6f2f6ab171160e355da74bf7b35737552afd4ee21f4c3afc0","name":"Screenshot 2022-09-21 at 9.53.39 PM.png","type":"image/png","lastModified":1663768461262.135,"size":2609954},{"id":"857787dc61da0fa23786997bb8eee11c0796490ecb5f17161825bbebf2626c9fa5451342c368ebe53676d73c109cff398a3023b61cea59e23c38056331707e82","href":"https://static.observableusercontent.com/files/857787dc61da0fa23786997bb8eee11c0796490ecb5f17161825bbebf2626c9fa5451342c368ebe53676d73c109cff398a3023b61cea59e23c38056331707e82","name":"Screenshot 2022-09-21 at 8.52.57 PM.png","type":"image/png","lastModified":1663764809054.854,"size":289504},{"id":"b69ec3e093a2ba19261a133832a937f0bfc7d6d29e681afd5fd7050a46425527a09dcd0436762f62a6b02a28ac5e7a22b2c66f39f12cd65e7178ea5800308792","href":"https://static.observableusercontent.com/files/b69ec3e093a2ba19261a133832a937f0bfc7d6d29e681afd5fd7050a46425527a09dcd0436762f62a6b02a28ac5e7a22b2c66f39f12cd65e7178ea5800308792","name":"Screenshot 2022-09-21 at 10.47.09 PM.png","type":"image/png","lastModified":1663771656381.893,"size":1248341},{"id":"54b96b6bda0c33cc6a3a32395f83d13fa358a413ca080b7302971a0eb3ee672f0776887230615210c77584b3ac9b6cbc7411108d56b310d2c8efe5d42c92f709","href":"https://static.observableusercontent.com/files/54b96b6bda0c33cc6a3a32395f83d13fa358a413ca080b7302971a0eb3ee672f0776887230615210c77584b3ac9b6cbc7411108d56b310d2c8efe5d42c92f709","name":"Screenshot 2022-09-21 at 9.53.47 PM.png","type":"image/png","lastModified":1663768460793.056,"size":1727408},{"id":"1456ff32c0b4744dec0738eea30588c940a9d18292c6ef59326b5e6d77c63115f4b20ff7fbb109765a85f52e5e57dbc20855570a6af62a8805620fa259773021","href":"https://static.observableusercontent.com/files/1456ff32c0b4744dec0738eea30588c940a9d18292c6ef59326b5e6d77c63115f4b20ff7fbb109765a85f52e5e57dbc20855570a6af62a8805620fa259773021","name":"Screenshot 2022-09-21 at 10.44.44 PM.png","type":"image/png","lastModified":1663771531223.681,"size":1227120},{"id":"991fde715ca4ef63547caaa4dc4077f2081b4645decabc541fc1d4461f35d95aba541c3cc0d41d66d7b6aaee0982d734b6b145137742b61f611eb8d216682229","href":"https://static.observableusercontent.com/files/991fde715ca4ef63547caaa4dc4077f2081b4645decabc541fc1d4461f35d95aba541c3cc0d41d66d7b6aaee0982d734b6b145137742b61f611eb8d216682229","name":"Screenshot 2022-09-21 at 10.44.57 PM.png","type":"image/png","lastModified":1663771531179.933,"size":1107629},{"id":"a204ee9cab5d0f0a1f13e4d45576f5885cc5e2569f92ccd8ccd3b66f64a9725d330b8a37d8eeeb0597501b6c7b9fa58fcd37d307da771d0a12d13997212949a7","href":"https://static.observableusercontent.com/files/a204ee9cab5d0f0a1f13e4d45576f5885cc5e2569f92ccd8ccd3b66f64a9725d330b8a37d8eeeb0597501b6c7b9fa58fcd37d307da771d0a12d13997212949a7","name":"Screenshot 2022-09-21 at 10.45.07 PM.png","type":"image/png","lastModified":1663771531221.768,"size":1179287},{"id":"8961d3a457f88dbd15f1c6ddb088d786f080dddc5cc1d90873cbc62d091354a658a967ca05f6de51d6ecb13ecbdb4a6d9460880a4747783bdeba9ba3b11a0c9b","href":"https://static.observableusercontent.com/files/8961d3a457f88dbd15f1c6ddb088d786f080dddc5cc1d90873cbc62d091354a658a967ca05f6de51d6ecb13ecbdb4a6d9460880a4747783bdeba9ba3b11a0c9b","name":"Screenshot 2022-09-22 at 1.15.40 AM.png","type":"image/png","lastModified":1663780651280.548,"size":1007414},{"id":"5dea94da114cbd2cb24aac17a9da28b96c597193392c63567a50e1ad2712dc70eacc9b0f9ccd5eae177701bc8e28f93f0503f88caccf3febef4ff6bd371b3d5a","href":"https://static.observableusercontent.com/files/5dea94da114cbd2cb24aac17a9da28b96c597193392c63567a50e1ad2712dc70eacc9b0f9ccd5eae177701bc8e28f93f0503f88caccf3febef4ff6bd371b3d5a","name":"Screenshot 2022-09-22 at 1.16.29 AM.png","type":"image/png","lastModified":1663780652244.02,"size":2041126}] 31:{"id":"ddce209f1ef4643e","slug":"learning-to-see-ml-project-presentation","title":"\"Learning To See\" - ML Project Presentation","thumbnail":"e99247c479085125ec5b8366a7f711e0b24e902f6aad026b178d078a8eeaf37b","access_level":"public","create_time":"$D2022-09-21T12:51:23.875Z","update_time":"$D2022-09-22T01:07:05.632Z","view_time":null,"first_public_time":"$D2022-09-21T17:25:19.331Z","has_unpublished_edits":null,"fork":false,"stars":2,"starred":false,"owner":{"type":"team","id":"c3d91d7db2867b4d","login":"betahsu","name":"ychsu6","avatar_url":"https://avatars-next.observableusercontent.com/197b7792c36f626b9fe391a472486b8b6dc3cf3503a2bb52e365e7916c324a9e","tier":"starter_2024"},"version":148,"publish_version":148,"role":null,"authored":false,"authors":[{"type":"individual","id":"8205ffe96d046f8c","login":"betahsu","name":"ychsu6","avatar_url":"https://avatars-next.observableusercontent.com/197b7792c36f626b9fe391a472486b8b6dc3cf3503a2bb52e365e7916c324a9e","tier":null}],"members":[{"type":"individual","id":"8205ffe96d046f8c","login":"betahsu","name":"ychsu6","avatar_url":"https://avatars-next.observableusercontent.com/197b7792c36f626b9fe391a472486b8b6dc3cf3503a2bb52e365e7916c324a9e","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":true,"trashed":false,"lapsed":null,"subscribed":null,"forks":1,"collection":null,"type":"notebook","body":{"title":"\"Learning To See\" - ML Project Presentation","cells":[{"id":0,"value":"# \"Learning To See\" - ML Project Presentation","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":5,"value":"$33","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":95,"value":"### Conceptual\n- Metaphor for the process of devloping a visual understanding of the world\n- Call to question how artificial neural networks & humans construct meaning\n- Goal: not necessarily to invent specific methods that accomplish certain tasks. But instead, to demonstrate the potential power of realtime, meaningful, continuous control over the outputs of generative Deep Neural Networks.\n \n### Interactive\n- To meaningfully control the aesthetics of the images generated by the Neural Network\n- Disable Real Time Learning cuz it's more difficult for a user to isolate and understand the creative impact of any actions that they might take, as the system is far more complex, with orders of magnitude more variables affecting the images generated.\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":141,"value":"### Technical\n#### System\n- Pre-trained(not real-time learning) model loaded to the real-time inference software\n- Software feeds each frame from a live video feed to the selected model\n- Display result in real-time\n\n(Difference between this system & Neural Style Transfer:
\nStyle Transfer extracts ‘style’ information from one or a few dozen images, but this model is trained on thousands of images; therefore, it contains information from across a much larger domain.)","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":138,"value":"$34","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":135,"value":"### Artworks applied this System\n#### Body Network on Paper I and II (2019) by Scott Eaton\n

\n#### Architectural machine translation (2019) by architect Erik Swahn\n

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":66,"value":"$35","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":144,"value":"## [Dirty Data (2017)](https://www.memo.tv/works/dirty-data/)\n- Images scraped from Google image search, of Donald Trump, Theresa May, Nigel Farage, Marine Le Pen, Recep Tayyip Erdogan and Vladimir Putin.\n- Not in anyway cleaned, cropped, filtered, aligned or sanitised. Not even checked if the images do indeed contain the subjects. Anything remotely related as deemed by Google image search is dumped into the model.\n\n
\n

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"}],"files":"$W36","stdlib":"1","titling":"auto"}} 32:"$31:body"