3f:null 40:["$","title",null,{"children":"Emoji in Javascript | Carlos Rotger | Observable"}] 46:Te71,md` ## Understanding strings in Javascript To talk about strings, first we have to talk about Unicode. Unicode is an attempt at creating a globally-accepted text encoding standard, to replace existing country-specific standards in use around the world, such as the American Standard Code for Information Exchange (ASCII). Each character is given a number called a "code point" (also occasionally known as a "rune"). Each code point is a 32 bit unsigned integer, often referenced in hex notation with leading zeroes omitted and a "U+": for instance U+0061 is the hex encoding of the number 97, which represents the character 'a'. This is exactly the same as the ASCII representation of 'a'. Other examples could be U+00ED, which is '${"\u00ed"}' ('i' with an accent), or U+2318 '${"\u2135"}', the Hebrew letter aleph, but interestingly it also includes many things that aren't letters, like U+2315 '${"\u2318"}'. The best examples of these are the emoji, originally a set of pictographs that shipped with Japenese mobile phones, but ended up staying for good in the unicode standard. The rainy day umbrella is U+2614, ${'\u2614'}. Although Unicode could in theory support many millions of code points, so far all that has been planned for is 0 through 10FFFF, and even that is only about 10% full. The simplest possible way to represent any Unicode code point in a computer would be to use a 32-bit unsigned integer for each character of a string. This would mean, however, that for "simple" Western text that largely uses the only the first 256 characters, files and strings would effectively be 4 times larger than they needed to be. And since only about 17 bits of information are in use, it would be generally needless space. So to get around this requirement, there are a series of variable-length encoding schemes to reduce the size of unicode texts. You may be familiar with UTF-8, which uses sequences of 8 bits each, but there's also UTF-16 and UTF-32, which use 16- and 32-bits per element respectively. ### UTF-32 UTF-32 is the simplest possible encoding. Each code point is represented by a single 32-bit unsigned integer, and so takes up 4 bytes of space. "Hello, World" == ${UTF32.html(toCodePoints("Hello, World", Uint32Array))} ### UTF-16 UTF-16 uses 16-bit chunks to represent code points. Generally, code points that are less than FFFF (16 bits) are simply represented by their actual value. "ABC" == ${UTF16Normal.html(toCodePoints("ABC", Uint16Array))} For values larger than FFFF, UTF-16 uses _two_ 16 bit chunks called a "surrogate pair". The first half of the pair begins with 110110, and the second with 110111, and the code point is split between the remaining bits in each pair after removing the leading 1 in the 17th bit. "🤑" = ${UTF16SurrogatePair.html(toCodePoints("🤑", Uint16Array))} "Money Mouth Face" is U+1F911, so the first element of the pair encodes 111110, and the second encodes 0100010001, which together is 0b11111100100010001 = 129297 = 0x1f911. ### UTF-8 UTF-8 uses 8-bit chunks to represent code points. If a code point uses 7 bits or less, it will take up only 1 byte. "a" = ${UTF8.html(toCodePoints("a", Uint8Array))} If a code point uses more than 7 bits, it is represented as a sequence of bytes. The first byte in the sequence indicates how many bytes there are, and the rest of the bytes start with an indicator that they're in a series. "${'\u00A2'}" = ${UTF8.html(Uint8Array.of(0xC2, 0xA2))} "${'\u20AC'}" = ${UTF8.html(Uint8Array.of(0xE2, 0x82, 0xAC))} "🤑" = ${UTF8.html(Uint8Array.of(0xF0, 0b10011111, 0b10100100, 0b10010001))} For English text and code, UTF-8 is usually the smallest representation of the text. `47:T7a6,md`## Other quirks with Emoji ### Variation Selector-16 Some emoji have two different renderings, one that looks more like text: ${'\u2663'}, and one that looks like a picture: ${'\u2614\uFE0F'}. To indicate which one of these should be used, a special invisible character called a "variation selector" is added to the end of the string. There are 16 of them (FE00-FE0F) but only the 16th seems to be in use. ### Modifier Sequences Some emoji are formed via a combination of an existing emoji and a modifier, to cut down on the number of code points issued. For instance, skin tone variations are formed using a "base" emoji like 👍 combined with a skin tone emoji: 🏽. So to form ${'\ud83d\udc4d\ud83c\udffd'}, we actually need four UTF-16 sequences, a surrogate pair for thumbsup, and the surrogate pair for medium-brown. In code, this would be "\\ud83d\\udc4d\\ud83c\\udffd". This has the advantage that on systems where skin tones haven't been implemented, we get a sort of half-version of what we're expecting: ${'\ud83d\udc4d \ud83c\udffd'}. ### Zero-Width Joiner Some emoji are formed via a combination of two others, also to cut down on the number of code points issued. For instance, some of the "profession emojis": 👩‍🔬 (female scientist) is 👩 (woman) and 🔬 (microscope) separated by U+200D, the zero-width joiner, AKA the "zwidge". 👨‍💻 is 👱‍♂️ (man) ZWJ 💻 (laptop). This can intersect with modifier sequences, so you can have 👩🏽‍🔬 (woman + medium tone ZWJ science), and 👨‍💻 is actually 👱‍ (person) modified by with ♂️ (male symbol) joined with 💻. My favorite are the "family" emojis, 👨‍👨‍👦‍👦 is actually 11 codepoints: man join man join boy join boy. These also function similarly in environments that don't support them -- if the join can't be made, it simply renders as everything next to each other: 👱‍♂️👱‍♂️👦👦. `48:T4a1,class BinDisplay { constructor(name = "", format = [4, 4, 8], labels = []) { this.format = format; this.name = name; const defaultStyle = ` `; this.style = defaultStyle; this.labels = labels; } static leftPadToLength(str, l, padding = "0") { return (padding.repeat(l)+str).slice(-l); } static getBinary(buf) { const bitsPerElt = (buf.constructor.BYTES_PER_ELEMENT * 8) || 8; const out = []; for (let byts of buf) { out.push(BinDisplay.leftPadToLength(byts.toString(2), bitsPerElt)); } return out; } toLabeledSpans(chunk) { let out = ""; for (let i = 0, idx = 0; i < this.format.length; idx += this.format[i], i += 1) { const label = this.labels[i] || ""; out += `${chunk.substring(idx, idx+this.format[i])}`; } return out; } html(data) { const stringForm = BinDisplay.getBinary(data); const spans = stringForm.map((s, i) => `${this.toLabeledSpans(s)}`).join(""); return html` ${spans} `; } }49:T44a,UTF8 = { const FirstByte1 = new BinDisplay("UTF8", [1, 7], ["leading 0", "ASCII value"]); const FirstByte2 = new BinDisplay("UTF8", [3, 5], ["indicates 2 byte sequence", "content"]); const FirstByte3 = new BinDisplay("UTF8", [4, 4], ["indicates 3 byte sequence", "content"]); const FirstByte4 = new BinDisplay("UTF8", [5, 3], ["indicates 4 byte sequence", "content"]); const RestOfBytes = new BinDisplay("UTF8", [2, 6], ["leading bytes", "content"]); class UTFChar extends BinDisplay { constructor() { super(); } html(data) { let firstByte; if (data.length == 1) { firstByte = FirstByte1.html(data); } if (data.length == 2) { firstByte = FirstByte2.html(data.slice(0, 1)); } if (data.length == 3) { firstByte = FirstByte3.html(data.slice(0, 1)); } if (data.length == 4) { firstByte = FirstByte4.html(data.slice(0, 1)); } const rest = RestOfBytes.html(data.slice(1)); return html` ${firstByte}${rest} `; } } return new UTFChar(); }4a:[] 44:{"id":"97d38719c2962be1","slug":"emoji-in-javascript","title":"Emoji in Javascript","thumbnail":"bafe9cc4b620bd43f0e8a98b6116d1eceae09194cbdd2189a9de3274fea0da36","access_level":"public","create_time":"$D2021-02-02T20:12:28.241Z","update_time":"$D2021-02-04T22:27:56.762Z","view_time":null,"first_public_time":"$D2021-02-04T20:05:20.989Z","has_unpublished_edits":null,"fork":false,"stars":1,"starred":false,"owner":{"type":"team","id":"2c7d546b70af5516","login":"crotger","name":"Carlos Rotger","avatar_url":"https://avatars-next.observableusercontent.com/196c68a196eac57c8ff54aaf5b1ca92b32f76d3a08344f5123102f37a8357351","tier":"starter_2024","bio":"","following":false},"version":607,"publish_version":607,"role":null,"authored":false,"authors":[{"type":"individual","id":"ef2bed8c473344ae","login":"crotger","name":"Carlos Rotger","avatar_url":"https://avatars-next.observableusercontent.com/196c68a196eac57c8ff54aaf5b1ca92b32f76d3a08344f5123102f37a8357351","tier":null,"bio":"","following":false}],"members":[{"type":"individual","id":"ef2bed8c473344ae","login":"crotger","name":"Carlos Rotger","avatar_url":"https://avatars-next.observableusercontent.com/196c68a196eac57c8ff54aaf5b1ca92b32f76d3a08344f5123102f37a8357351","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":2,"collection":null,"type":"notebook","body":{"titling":"auto","title":"Emoji in Javascript","cells":[{"id":0,"value":"md`# Emoji in Javascript`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":374,"value":"$46","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":513,"value":"$47","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":543,"value":"md`# Tips For JS\n## 1. String length is wonky\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":546,"value":"\"☔️\".length","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":548,"value":"\"👨‍👨‍👧‍👦\".length","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":553,"value":"md`## 2. Use codePointAt, not charCodeAt`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":556,"value":"\"🤑\".codePointAt(0).toString(16)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":557,"value":"\"🤑\".charCodeAt(0).toString(16)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":568,"value":"md`## 3. Account for surrogate pairs`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":584,"value":"\"wow! 🤑 i made so much money 💸 using this app?? 📲\".match(/([\\u10000-\\u1FFFFF])/g)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":571,"value":"\"wow! 🤑 i made so much money 💸 using this app?? 📲\".match(/([\\ud800-\\udbff][\\udc00-\\udfff])/g)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":589,"value":"md`## 4. Watch out for modifiers`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":590,"value":"\"☔️\" == \"☔️️\" // one of these has a VAR-16 at the end!","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":596,"value":"\"I'm saving for 👩🏽‍🔬\".match(/([\\ud800-\\udbff][\\udc00-\\udfff])/g)[0] // not what they meant!","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":329,"value":"html`\n\n`","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":449,"value":"toCodePoints = function(str, TypedArray) {\n return TypedArray.from(str.split('').map(c => c.codePointAt(0)))\n}","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":6,"value":"$48","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":7,"value":"ASCII = {\n let style = ``;\n return new BinDisplay(\"ASCII\", [1, 7], [\"leading 0\", \"7-bit number\"]);\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":214,"value":"$49","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":335,"value":"UTF16Normal = new BinDisplay(\"UTF16Normal\", [16], [])","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":339,"value":"UTF16SurrogatePair = new BinDisplay(\"UTF16Pair\", [6, 10], [\"pair indicator\", \"content\"])","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":340,"value":"UTF16SurrogatePair.html(Uint16Array.of(500, 499))","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":342,"value":"UTF32 = new BinDisplay(\"UTF32\", [32], [])","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":69,"value":"ASCII.html(Int8Array.of(64, 63, 43))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":106,"value":"html`☔️ ${ASCII.html(Uint8Array.of(1, 2))}`","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":253,"value":"html`\n\n${UTF8.html(Uint8Array.of(237, 175, 180))}\n`","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W4a","stdlib":"1"}} 45:"$44:body"