2b:null 2c:["$","title",null,{"children":"Asks (by Piece Size) [Provider.Quest] | Jim Pick | Observable"}] 31:Te87,md`Several times per day, I am scanning all the storage providers on Filecoin that appear to be reachable from my location the Internet, and I then query their "ask" information using [@jimpick/provider-quest-storage-ask-scanner](https://observablehq.com/@jimpick/provider-quest-storage-ask-scanner?collection=@jimpick/provider-quest). The latest asks are combined into a JSON report and published to a Textile Bucket. The JSON file can be fetched via HTTP ... the [@jimpick/provider-quest-feeds](https://observablehq.com/@jimpick/provider-quest-feeds?collection=@jimpick/provider-quest) notebook has the URLs and exports for fetching this data, and several other useful feeds. You can use the tool in this notebook to select a piece size (the data size you want to store, rounded up to a power of 2), and view the miners that support storing that size, ordered by price (either non-verified or verified a.k.a. Filecoin Plus). **Historical Deal Data:** To help put the pricing in context, I am now also showing historical published deal data. The data shows the number of deals per miner, the total size of the deals combined together, and the number of distinct clients over a window of ${multidayDealsReport.intervalDays} days. I'm showing statistics aggregated for all piece sizes, not just the selected piece size. Historical deals may have been accepted at different price points than current asks. **Historical Power Data:** I am now also showing historical power data. The data shows the power acquired by a miner averaged over a window of ${multidayDealsReport.intervalDays} days. The historically averaged power data should be less volatile than live power data which can periodically dip to zero when miners encounter transient issues. **Legacy Testing Annotations:** For now, I am also displaying some annotations for each miner which shows some results from my "legacy" testing system with very small file sizes. The annotation states only represent the result of my latest round of testing and are not necessarily up-to-date or reliable. I am working to replace this system with a better one. For each miner, I make a crude attempt to geo-locate them based on their IP address, but I don't usually update these often. My future testing system will automate this better. I use the label *"NR"* (non-routable) when I couldn't find an IP address (but the miner may have added one after I made the annotation). Miners tagged as **"active"** are ones which I was able to successfully store a small file and for which the miner has completed the proof of replication and is proving the storage to Filecoin. Other annotations you might see are **"sealing"**, or **"active-sealing"**, which means the miner has accepted my deals, but hasn't finished sealing yet. You might see **"inflight"**, or **"stuck"** when I've sent a deal but the miner hasn't started sealing it yet. Some deals end in **"error"** due to many different problems - you can see some error logs as comments on [GitHub](https://github.com/jimpick/workshop-client-mainnet/blob/main/src/annotations-mainnet.js). Often the error is on my end of the deal as I am frequently upgrading my Lotus node and running unstable versions. If you see **"min-size"**, it is because I'm only testing small files and the miner only accepts large files. If you see **"min-ask"**, it's because my legacy testing system is unaware of ask pricing, and sends deals to miners with a lower deal price than they are willing to accept. Some miners rejected my deals for various reasons, so those are labeled as **rejected**. I'm filtering out miners that I can't dial. I am going to replace this testing system with a smarter one built using ObservableHQ notebooks.`32:T667,{ const titles = 'Miner | ' + (sortBy === 'Price' ? 'Price | Verified Price' : 'Verified Price | Price') + ' | Deals | Total Size | Clients | Power | Annotation' const output = md`${titles} ---|---|---|---|---|---|--- ${filteredAsks.sort(sortByFunc).map(({ miner, price, verifiedPrice }) => { let line = `${miner} | ` + (sortBy === 'Price' ? `${price} | ${verifiedPrice}` : `${verifiedPrice} | ${price}`) + ` | ` + (multidayDeals[miner] ? multidayDeals[miner].count : '') + ` | ` + (multidayDeals[miner] ? bytes(multidayDeals[miner]['sum(pieceSizeDouble)'], { mode: 'binary' }) : '') + ` | ` + (multidayDeals[miner] ? multidayDeals[miner]['approx_count_distinct(client)'] : '') + ` | ` + (minerPowerMultidayReport && minerPowerMultidayReport.miners[miner] ? bytes(minerPowerMultidayReport.miners[miner].qualityAdjPower, { mode: 'binary' }) : '') + ` | ${legacyAnnotationsMainnet[miner]}\n` return line }).slice(0, maxRows)} ` return output function sortByFunc(a, b) { let compare if (sortBy === 'Price') { compare = a.priceDouble - b.priceDouble } if (sortBy === 'Verified Price') { compare = a.verifiedPriceDouble - b.verifiedPriceDouble } if (compare !== 0) return compare const sizeA = (multidayDeals && multidayDeals[a.miner]) ? multidayDeals[a.miner]['sum(pieceSizeDouble)'] : 0 const sizeB = (multidayDeals && multidayDeals[b.miner]) ? multidayDeals[b.miner]['sum(pieceSizeDouble)'] : 0 const compareSize = sizeB - sizeA if (compareSize !== 0) return compareSize return Number(a.miner.slice(1)) - Number(b.miner.slice(1)) } }33:T678,filteredAsks = { const entries = Object.entries(asksSubsetLatest.miners) .map(([miner, { timestamp, ...rest }]) => ([miner, { timestamp: d3.isoParse(timestamp), ...rest }])) .filter(([miner, ask]) => ( ask.timestamp >= minTimestamp && ask.minPieceSize <= selectedSize.bytes && ask.maxPieceSize >= selectedSize.bytes && (filterOut.includes('ridiculous price') ? ask.priceDouble < 1e14 : true) && (filterOut.includes('ridiculous verified price') ? ask.verifiedPriceDouble < 1e14 : true) && (filterOut.includes('rejected') && legacyAnnotationsMainnet[miner] ? !legacyAnnotationsMainnet[miner].match(/^rejected,/) : true) && (filterOut.includes('no deals') ? (multidayDeals && multidayDeals[miner]): true) && (filterOut.includes('<10 deals') ? (multidayDeals && multidayDeals[miner] && multidayDeals[miner].count >= 10): true) && (filterOut.includes('<50 deals') ? (multidayDeals && multidayDeals[miner] && multidayDeals[miner].count >= 50): true) && (filterOut.includes('<100 deals') ? (multidayDeals && multidayDeals[miner] && multidayDeals[miner].count >= 100): true) && (filterOut.includes('<3 clients') ? (multidayDeals && multidayDeals[miner] && multidayDeals[miner]['approx_count_distinct(client)'] >= 3): true) && (filterOut.includes('<5 clients') ? (multidayDeals && multidayDeals[miner] && multidayDeals[miner]['approx_count_distinct(client)'] >= 5): true) /* && !legacyAnnotationsMainnet[miner].match(/^fail,/) && !legacyAnnotationsMainnet[miner].match(/^delist,/) */ )) .map(([miner, ask]) => ({miner, ...ask})) return entries }34:[] 2f:{"id":"3a823885ba424869","slug":"provider-quest-piece-size-vs-asks","title":"Asks (by Piece Size) [Provider.Quest]","thumbnail":"7a1cf79ed4eb0758594ffc5fd6c900f78bb8cea653f53150153e321a7816b63a","access_level":"public","create_time":"$D2021-06-17T02:33:56.012Z","update_time":"$D2022-03-27T02:07:46.912Z","view_time":null,"first_public_time":"$D2021-06-17T03:05:12.721Z","fork":false,"stars":0,"starred":false,"owner":{"type":"team","id":"5252cea433451319","login":"jimpick","name":"Jim Pick","avatar_url":"https://avatars-next.observableusercontent.com/e8813527082139d5411445010e72fd85f99b04a26355e795d0ea26765425d0cd","tier":"starter_2024"},"version":540,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"bda4505ba9a55ac8","login":"jimpick","name":"Jim Pick","avatar_url":"https://avatars-next.observableusercontent.com/e8813527082139d5411445010e72fd85f99b04a26355e795d0ea26765425d0cd","tier":null}],"members":[{"type":"individual","id":"bda4505ba9a55ac8","login":"jimpick","name":"Jim Pick","avatar_url":"https://avatars-next.observableusercontent.com/e8813527082139d5411445010e72fd85f99b04a26355e795d0ea26765425d0cd","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"subscribed":null,"forks":0,"collection":{"id":"242749e359cb986b","slug":"provider-quest","title":"Provider.Quest","owner":{"id":"5252cea433451319","login":"jimpick","name":"Jim Pick"}},"type":"notebook","body":{"title":"Asks (by Piece Size) [Provider.Quest]","cells":[{"id":0,"value":"# Asks (by Piece Size) [Provider.Quest]","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":493,"value":"md`${quickMenu}`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":102,"value":"$31","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":26,"value":"viewof selectedSize = Inputs.select(sizesBytes, { format: x => x.pretty, label: \"Piece Size\", value: sizesBytes.find(x => x.pretty === '1GiB') })","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":96,"value":"md`Prices are in attoFIL per epoch (30 seconds) per GiB`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":66,"value":"viewof sortBy = Inputs.radio(['Price', 'Verified Price'], {label: \"Sort By\", value: 'Price'})","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":134,"value":"viewof filterOut = Inputs.checkbox([\"ridiculous price\", \"ridiculous verified price\", \"rejected\", \"no deals\", \"<10 deals\", \"<50 deals\", \"<100 deals\", \"<3 clients\", \"<5 clients\"], {label: \"Filter out\", value: [\"ridiculous price\", \"ridiculous verified price\", \"no deals\", \"<10 deals\", \"<50 deals\", \"<3 clients\"] })","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":57,"value":"$32","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":293,"value":"maxRows = 1000","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":363,"value":"md`## Price Converter Widget`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":385,"value":"viewof selectedPrice = Inputs.select(pricePoints, { format: x => `${x} attoFIL`, label: \"Price Per Epoch\" })","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":389,"value":"{\n const filPerEpoch = Number(selectedPrice) / 1e18\n const epochsPerYear = 365 * 24 * 60 * 2\n const sizeInGiB = selectedSize.bytes / 1024 ** 3\n const filCost = filPerEpoch * epochsPerYear * sizeInGiB\n const usdCost = filCost * filUsdFromCoinGecko\n return md`Cost to store ${selectedSize.pretty} for 1 year = **${filCost.toFixed(4)} FIL** ($${usdCost.toFixed(4)} USD\\*)
Note: Using current trading price from the CoinGecko API: 1 FIL = $${filUsdFromCoinGecko} USD`\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":150,"value":"md`## Data`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":4,"value":"sizes = [...Array(29).keys()].map(i => 2 ** (8 + i))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":11,"value":"sizesBytes = sizes.map(size => ({ pretty: bytes(size, { mode: 'binary' }), bytes: size }))","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":39,"value":"import { asksSubsetLatest } from '@jimpick/provider-quest-feeds'","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":87,"value":"asksSubsetLatest","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":522,"value":"liveAsks = {\n const entries = Object.entries(asksSubsetLatest.miners)\n .map(([miner, { timestamp, ...rest }]) => ([miner, {\n timestamp: d3.isoParse(timestamp),\n ...rest\n }]))\n .filter(([miner, ask]) => (\n ask.timestamp >= minTimestamp\n ))\n .map(([miner, ask]) => ({miner, ...ask}))\n return entries\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":468,"value":"minTimestamp = dateFns.subDays(new Date(), 3)","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":41,"value":"$33","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":55,"value":"import { legacyAnnotationsMainnet } from '@jimpick/provider-quest-feeds'","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":61,"value":"legacyAnnotationsMainnet","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":325,"value":"import { dealsBucketUrl } from '@jimpick/provider-quest-feeds'","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":268,"value":"multidayDealsReport = (await fetch(`${dealsBucketUrl}/multiday-average-latest.json`)).json()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":287,"value":"multidayDealsReport","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":270,"value":"multidayDeals = multidayDealsReport.providers","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":329,"value":"import { minerPowerMultidayAverageLatestBucketUrl } from '@jimpick/provider-quest-feeds'","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":332,"value":"minerPowerMultidayReport = (await fetch(`${minerPowerMultidayAverageLatestBucketUrl}/miner-power-multiday-average-latest.json`)).json()","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":373,"value":"pricePoints = {\n const prices = new Set()\n for (const ask of filteredAsks) {\n prices.add(ask.price)\n prices.add(ask.verifiedPrice)\n }\n return [...prices].sort((a, b) => Number(a) - Number(b))\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":366,"value":"filUsdFromCoinGecko = (await (await fetch('https://api.coingecko.com/api/v3/simple/price?ids=filecoin&vs_currencies=usd')).json()).filecoin.usd","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":12,"value":"md`## Imports`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":18,"value":"bytes = (await import('@jimpick/bytes-iec@3.1.0-2')).default","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":461,"value":"d3 = require(\"d3@6\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":465,"value":"dateFns = require('https://bundle.run/date-fns@2.22.1')","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":489,"value":"import {quickMenu} from '@jimpick/provider-quest-utils'","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":527,"value":"## Backups","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":531,"value":"import {backups, backupNowButton} from '@jimpick/provider-quest-utils'","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":534,"value":"backups()","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"}],"files":"$W34","stdlib":"1","titling":"auto"}} 30:"$2f:body"