2d:null 2e:["$","title",null,{"children":"SM3809_A11 | John C | Observable"}] 33:T273d,html`

SM3809_A11

AI & Critical Practice

Reading

READ_ME , RUN_ME, EXECUTE_ME : SOFTWARE AND ITS DISCONTENTS, OR : IT’S THE PERFORMA- TIVITY OF CODE, 1 STUPID by Inke Arns:

Differences between generative art and software art: generative process; digital/non-digital; result/process, structure; media/material
software art as a kind of investigation/impact
question on/utilize software
the software itself (executed or not) as the artwork
performativity of software: concrete actualization, code is law / non-executability — i.e. dysfunctionality / protentially executable
potentially totalitarian dimension of the algorithmic program code

Review

DCGAN

GAN: The idea of Generative Adversarial Network (生成對抗網路) is first introduced by Ian J. Goodfellow et al. for unsupervised learning. To put it simply, there are two parts in GANs: a generator(G) that generate data(usually images) and a discriminator(D). The purpose of the generator(G) is to generate "realistic" data to fool the discriminator(D); while the purpose of the discriminator(D) is to distinguish fake data. In short, we can imagine G as a cash forger and D as a government agent(police), the learning process of GANs is like the fight between the forger and the police: the forger learn to forge better, and as a result, the police have to learn new skills to distinguish the fake cash, which forces the forger to learn new tricks, and the process repeats when both of them obtain better skills. In most of the projects using GANs, G and D are both deep learning networks.

In practice, G and D are not so dynamic as the forger and the police in the metaphor. Normally, in phase 1, we start with a 'not bad' D and train G with it first. As G getting better and better with the training, it starts to be able to fool D. When D starts to give 50:50 results, it means that D is fooled and starts to guess randomly. Now it is meaningless to keep training G since D cannot give useful feedback, so we enter phase 2, in which we fix G and train D with it. With the training, D gets better and better. When D can point out all the fake data, this phase is ended. Then we repeat the two phases until G can generate very realistic results.

https://thispersondoesnotexist.com/: an example of how GAN can create super realistic portraits, this site can also be used as an api (I have used it serval time and found that it is kind of good because the since portrait is generated, there is no portrait rights issue)

Advantages of GANs:

unsupervised learning, which means we don't need to label tons of things
theoretically, GANs can be used to train every kind of neural network

Disadvantages of GANs:

not easy to make G and D work together (if we did not design the training well, often at the end G starts to create random data, and D converges to simple rules)
Mode Collapse: in the learning process, there is also a chance that at some point G starts to generate repeat data, which stops G and D from moving forward.

DCGAN: Deep Convolutional Generative Adversarial Network (深度卷積生成對抗網路). DCGAN is a kind of GANs in which convolution is introduced into D and deconvolution is introduced into G.

In short, convolution is the process of reducing the dimensions of data. Let's take image processing as an example. Imagine now we have a standard HD video frame, which is an image of 1920*1080 pixels, and each pixel contains 3 colors(RGB). In this case, the amount of data to be processed is huge (1920*1080*3 = 6220800 numbers ). But think about that: as human beings, we can process images in much much higher resolution(the resolution of our eyes is about 576000000 pixels, see this article) in super high framerate (the human eyes are able to recognize a picture in 1/200 second). How do we do that? Well, this is a complex topic in human vision study, but a short possible answer is, we do not process every 'pixel', but try to get the 'features' of the image. David Hubel and Torsten Wiesel (two Nobel Prize Winners) discovered that information from our eyes is processed through several layers in our brain. In the lower layers, our brains found edges and directions in the image; then in higher layers, the image is abstracted more and more advanced features are recognized.

Convolution networks are based on the same idea. A network with multiple layers is set up to imitate the process in our brain. Normally the network contains three parts: convolutions, sub-sampling (pooling), and full-connection. In production, there are often multiple loops of convolutions and sub-sampling.

Convolution is a very common technique in digital image processing. The process of convolving an image looks like this.

In the simple case above we 'compress' a 5*5 picture to a 3*3 'feature set'. In real-world practice, people often use multiple convolution matrices designed for different patterns to get different features of the image.

Because the size of convolution matrices is not big (in many cases, it's just 3*3 or 5*5), so the size of data might be still too big after convolutions. So we need a sub-sampling(pooling) process after convolutions. Below is a gif that visualizes the process.

In the sub-sampling process, a bigger sample window is used to reduce the size(or the number of dimensions) of the data.

After going through several loops of convolutions and sub-sampling, the size of data is reduced to an ideal range, so the final part is (kind of) the 'real' neural net, the fully connectioned part. In this part, the pre-processed data are put in the input layer, and after going through (lots of) hidden layers, we got the output from the output layer.

We can think about convolution neural networks, in a way, as ordinary neural networks plus some steps of data pre-processing, in order to handle data in huge size. So, it is important to make sure that the information loss in the 'pre-processing steps' is at the minimum level, which requires careful design of the convolutions and sub-sampling parts.

Back to DCGAN, its main difference from other GANs is that the above pre-process steps(convolutions and sub-sampling) are used in the discriminator(D), and the reverse process of the steps(which is called 'deconvolution') is applied in the generator(G). This difference allows DCGAN to handle data in huge size(e.g images) well.

To use DCGAN with ml5.js:


// to use the pre-trained model
const dcgan = ml5.DCGAN('path/to/model', modelReadyCallback);

const modelReadyCallback = function(){
  dcgan.generate(imageCallback);
}

const imageCallback = function(err, result){
  if (err) throw Error("image generate error: " + err.message);
  // do something with the result... e.g download the image
  downloadBlob(result.blob, 'new_gerneration.png');
}

const downloadBlob = function(blob, name = 'image.png') {
    if (window.navigator && window.navigator.msSaveOrOpenBlob) 
       return window.navigator.msSaveOrOpenBlob(blob);
    const data = window.URL.createObjectURL(blob);
    const link = document.createElement('a');
    link.href = data;
    link.download = name;
    link.dispatchEvent(
      new MouseEvent('click', { 
        bubbles: true, 
        cancelable: true, 
        view: window 
      })
    );
    setTimeout(() => {
      window.URL.revokeObjectURL(data);
      link.remove();
    }, 100);
}
//note: the model needs sometime to load...
//so not a good idea to load the model when the user click 'generate'
//and the result is below... (not very good I would say)

If you want to train your own models to use in ml5.js, visit this repo. Note that the training use Python instead of JavaScript.

Idea of an artwork:
The affordance of DCGAN is that DCGAN can generate realistic but fake things. So I think DCGAN can be useful in creating fake news. A possible project can be obtaining data from big newspapers in Hong Kong and create an news page for fakenews only with DCGAN (www.fakenewshk.com or something like that...)

above is a refernce of this kind of websites, World News Daily Report

Submit

Links:
Leung Man Tao: "Naai Gong" or "Gang Du"
In Liz's

`34:[{"id":"e8ab08ece3d876f8f0061973cf8ab774975ec1cbe9ca1e41ef388016bb40540e3154568ec6ea8202eaf7cf51e3694bebff6ad4c4cda79db80a137dc12620e9e5","href":"https://static.observableusercontent.com/files/e8ab08ece3d876f8f0061973cf8ab774975ec1cbe9ca1e41ef388016bb40540e3154568ec6ea8202eaf7cf51e3694bebff6ad4c4cda79db80a137dc12620e9e5","name":"1*z7hd8FZeI_eodazwIapvAw.png","type":"image/png","lastModified":1618395396314,"size":54775},{"id":"5a738508fe9ccd3be63d1444da98703c11ccb273bafb906ff9fe0b01a01e797e3342758e6173cb487485a3fff39b8300a09922a17a2a9ead00ecdef77ad1b75e","href":"https://static.observableusercontent.com/files/5a738508fe9ccd3be63d1444da98703c11ccb273bafb906ff9fe0b01a01e797e3342758e6173cb487485a3fff39b8300a09922a17a2a9ead00ecdef77ad1b75e","name":"螢幕截圖 2021-04-14 下午8.33.52.png","type":"image/png","lastModified":1618403666124.973,"size":3189837},{"id":"b125026348b42d45f1ee42fe37456e7aa323b9b8995e5cfd7f3ef41d8df617df1e076b458d8521aa72aa8f7d05f5a44bdd8dff5688dc595a668595d5d44afbf9","href":"https://static.observableusercontent.com/files/b125026348b42d45f1ee42fe37456e7aa323b9b8995e5cfd7f3ef41d8df617df1e076b458d8521aa72aa8f7d05f5a44bdd8dff5688dc595a668595d5d44afbf9","name":"螢幕截圖 2021-04-14 上午12.00.28.png","type":"image/png","lastModified":1618329658735.08,"size":265342}] 31:{"id":"d27d7711f988b560","slug":"sm3809-a11","title":"SM3809_A11","thumbnail":"ee85590c1fadee00afc0895b734adc94f283b698ac43be2eaa85c2fdd40f77b5","access_level":"public","create_time":"$D2021-04-11T07:11:58.891Z","update_time":"$D2021-04-15T11:02:53.443Z","view_time":null,"first_public_time":"$D2021-04-13T06:28:59.749Z","has_unpublished_edits":null,"fork":false,"stars":0,"starred":false,"owner":{"type":"team","id":"b2e9840be7b1b607","login":"real-john-cheung","name":"John C","avatar_url":"https://avatars-next.observableusercontent.com/1d9ef6121a0bbd1a75fab479f9891fe1d78cc65f76c394585ae1a7dad0d3abf6","tier":"starter_2024"},"version":111,"publish_version":111,"role":null,"authored":false,"authors":[{"type":"individual","id":"e8e99acdbe3aebfb","login":"real-john-cheung","name":"John C","avatar_url":"https://avatars-next.observableusercontent.com/1d9ef6121a0bbd1a75fab479f9891fe1d78cc65f76c394585ae1a7dad0d3abf6","tier":null}],"members":[{"type":"individual","id":"e8e99acdbe3aebfb","login":"real-john-cheung","name":"John C","avatar_url":"https://avatars-next.observableusercontent.com/1d9ef6121a0bbd1a75fab479f9891fe1d78cc65f76c394585ae1a7dad0d3abf6","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"title":"SM3809_A11","cells":[{"id":0,"value":"$33","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":9,"value":"html`
`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":12,"value":"html`\n