31:T8fd,md`## Technical details 1. There are 3 databases used and created by the back-end system: a "raw" database that contains recently deleted domains from ExpiredDomains.net; a "TBC" database that contains domains that are checked to be dead, waiting for the final check 12 hours later; a "graveList" database that contains domains that pass two rounds of if-is-dead check and information about the domains from archives. 2. The method used to check if a website is dead: send http request to the domain to see if it's online now, if it's not online, search the archives to see if there is any record of it (to prove that it once existed), if so, that we think this website is dead. Only websites that "pass" two rounds of the if-is-dead test 12 hours apart will be included in the graveyard. 3. When a website is confirmed to be dead, it will be put into the "graveList", and the Twitter bot will announce it on Twitter. The website is generated from the "graveList" on the client-side, i.e. each time the website is loaded in your browser, it will fetch the "graveList" from the database and construct the website according to the data. So disabling javascript will prevent the website from functioning. 4. For each website, the fingerprint of it is generated by a hash algorithm from its content (i.e. snapshots in the archives). Thus, the fingerprint is unique. The similarity of the first snapshot and the last snapshot is calculated in a similar way: two hash values are generated from the two snapshots to be compared. The reason for this to be able to handle different (and we don't know what) languages. 5. For more details that are not covered here, please refer to the code comments (github repo). 6. Reference: using data from ExpiredDomains.net; using Timetravel API; using npm libraries including: axios v0.21.1, crypto-js v4.0.0, iconv-lite v0.4.24, jsdom v16.5.3, langdetect v0.2.1, minhash v0.0.9, natural v5.0.3, novel-segment v2.7.58, parse v3.2.0, rita v2.4.86, twitter-lite v1.1.0; using browser js libraries including: jQuery v3.6.0, wordcloud2 v1.2.2. `32:T9ea,md`## Development Logs ### Note 6/May Navigate around the found record? Got images ? Geolocation from historical IP address? Who owns the domain previously? Subdomains as the families? Whowas apnic.net Ns lockup history whoisXML API Mongo db More info form the archived file. ### Note 8/May Better analyzer with better words extract function and better sentiments Force load font Static rawpool from expiredomain.net (not sure if it is a great idea to crawl info from it coz it's violating the terms. ### Note 10/May Sadly there is a bug preventing Firebase SDK from working with Node V14 and esm... (DO STH GOOGLE!) [Github issue](https://github.com/firebase/firebase-js-sdk/issues/4241) After one whole day trying to find a fix, I think instead of rewriting the whole project with CommonJS, maybe it's wiser to change to other database services ... :( Or is it? Cry cry trying with heroku... ### Note 11/May Using postgreSQL with heroku (with node.js support and hopefully accessible from client side with [postgREST](https://github.com/PostgREST/postgrest)) update: Not enough free dynos may need to turn to AWS or something like that... :( ### Note 12/May decided to use back4app for the database (it has built-in REST support and js api) and still hosting the bot on heroku... The databse connection seems to be ok better web design for cross-device performance & responsitivity better icon idea? It turns out that old pages have tons of messy problems (like strange charset and outdated syntax ...) need some time to clean them ### Note 13/May re-design the website (word cloud, barcode, more reactivity...) && test with more data (and debug... a lot) the limited request of the free DB running out quite fast :( so for now can just use local files for the design & tests (and upload everything on server/DB when all things are ready) ### Note 14/May add twitter bot debug and imporve algorithms add page similarity via min-mash map still thinking about what text/visual that sentiment can transform to... ### Note 15/May local test past, ready to test with back4app database Still, the content and visual on the website and the tweets need to be re-evaluated. ### Note 17/May add request timeout for preventing being hung forever adjust http requests flow & add about page for the website ### Note 18/May all local test runs well, deploy the bots on heroku ### Note 19/May fix bugs on heroku ### Note 20/May test website on mobile devices & fix some layout problem & optimize for Safari `33:[] 2f:{"title":" Internet Graveyard Logs","cells":[{"id":0,"value":"html`

Internet Graveyard Logs

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":12,"value":"md`Internet Graveyard is the place where the dead websites rest\n\nA Twitter bot will work as a gravedigger, every 12 hours it will find a webpage that had been somewhere online but no longer accessible (aka, a webpage that is dead), and make a grave for it with its digital fingerprint.\n\nAll the graves it makes will be displayed on a [website](https://real-john-cheung.github.io/Internet-Graveyard/), the Internet Graveyard.\n\ninspire by project [/death/null](http://www.deathnull.org/?what=about) by Ulf Schleth`","pinned":false,"hidden":false,"mode":"ojs"},{"id":15,"value":"html`Github repo: here
\nWebsite: here
\nTwitter Bot: here`","pinned":false,"hidden":false,"mode":"ojs"},{"id":113,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":25,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":19,"value":"html`

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":20,"value":"html`\n