29:["$","title",null,{"children":"Repression data API | Daniil Beilinson | Observable"}] 2f:T461,{ const schema = await getData('schema') const shortSchema = Object.keys(schema).reduce((acc, entityId) => { if (!acc[entityId]) acc[entityId] = { type: entityId, references: [], backreferences: [], props: Object.keys(schema[entityId]).length } acc[entityId].references = acc[entityId].references.concat(Object.values(schema[entityId]).filter(field => field.reference && field.dataType.indexOf('reference') > -1).map(field => field.reference)) acc[entityId].backreferences = acc[entityId].backreferences.concat(Object.values(schema[entityId]).filter(field => field.reference && field.dataType.indexOf('backreference') > -1).map(field => field.reference)) return acc }, {}) return dot`digraph { graph [label="Relations between entities in dataset (with number of properties describing entity)",fontsize=10,labeljust=l]; node [shape=box,fontsize=10]; ${Object.values(shortSchema).map(entity => ` "${entity.type}" [label="${entity.type}\n (${entity.props})"] ${entity.references.map(ref => ` "${entity.type}" -> "${ref}" `).join('\n')} `).join('\n')} }` }30:T5b8,The flat data API presents flat data according to the schema above. As with the schema, the data available in 2 languages and the fields included in different subsets. Basic subset contains only basic attributes, whereas detailed and full containing more data. The data can be requested by GET query using this URL: https://api.repression.info/v1/data. Available options through url query params: 1. language (`ru` or `en`), with default value `ru` 2. subset (`basic`, `detailed` or `full`), with default value `basic` 3. filter - data filters in format of JSON object (for example `{"persecution_minor":true,"persecution_started_year":[2022,2023]}` will return only persecution of minors started in 2022 and 2023), note that for a moment you can only filter data based on specific attribute value (e.g. `{"persecution_minor":true}` or by match with one of the values `{"persecution_started_year":[2022,2023]}`) 4. preset (_antiwar_ will filter only records related to persecutions for public antiwar stance, _actual_ will filter records related to actual persecutions, _imprisoned_ will filter records with imprisoned people, _after-invasion_ will filter records related to persecution started on 24/02/2022 or afterwards, _before-invasion_ will filter records related to persecution started before 24/02/2022) 5. csv (if `true` - will return data in csv file) As a result, you will get a JSON object with total number of matching records and data:31:T825,A listing of data can be requested by GET query using this URL: https://api.repression.info/v1/[entityCollection]. Available options through url query params: 1. language (`ru` or `en`), with default value `ru` - language of data 2. fields - an array with specific fields you want to request 3. subset (`basic`, `detailed` or `full`), with default value `basic` - subset of fields (see schema above) 4. references (`title`, `basic`, `detailed` or `full`), with default value `title` - controls level of exposed details within referenced entities (e.g. `title` will return only title of reference, whereas other option allows receiving more detailed data), note that nested references will not be fetched 5. filter - data filters in format of JSON object (for example `{"persecution_minor":true,"persecution_started_year":[2022,2023]}` will return only persecution of minors started in 2022 and 2023), note that for a moment you can only filter data based on specific attribute value (e.g. `{"persecution_minor":true}` or by match with one of the values `{"persecution_started_year":[2022,2023]}`) 7. preset (_antiwar_ will filter only records of entity _case_ related to persecutions for public antiwar stance, _actual_ will filter records of entities _persecution_ and _person_ related to actual persecutions, _imprisoned_ will filter records of entities _persecution_ and _person_ with imprisoned people, _after-invasion_ will filter records of entity _persecution_ related to persecution started on 24/02/2022 or afterwards, _before-invasion_ will filter records of entity _persecution_ related to persecution started before 24/02/2022) - preset of filters As a result, you will get a JSON object with total number of matching records and data, where each record has the following structure: 1. id - _id of record_ 2. type - _type of entity_ 3. attributes - _object with entity attributes_ 4. references - _information about references to the other entities_ Here is how the result of the request will look like for listing of _persons_ with _full_ preset and _antiwar_ subset:32:T51d,The data of particular entity can be requested by GET query using this URL: https://api.repression.info/v1/[entityCollection]/[entityId]. Available options through url query params: 1. language (`ru` or `en`), with default value `ru` - language of data 2. fields - an array with specific fields you want to request 3. subset (`basic`, `detailed` or `full`), with default value `basic` - subset of fields (see schema above) 4. references (`title`, `basic`, `detailed` or `full`), with default value `title` - controls level of exposed details within referenced entities (e.g. `title` will return only title of reference, whereas other option allows receiving more detailed data), note that nested references will not be fetched Note that as in case with the response to listing request, by default attributes will contain titles of referencing entities according to the selected subset. E.g. _full_ subset will contain information about all references and will inject all the titles of referenced data into the result. This can be changed with _references_ attribute. You can override this behaviour, querying for specific fields with _fields_ parameter. Take a look how the data will change for this entity describing persecution of _Alexey Gorinov_ depending on language, subset and references values:33:[] 2c:{"id":"3dfc2fcf987505cb","slug":"repression-data-api","title":"Repression data API","thumbnail":"45463efa99b5b704c81e0127508354281caae4b2ad9914f211e573a67538b461","access_level":"public","create_time":"$D2023-10-11T12:57:38.429Z","update_time":"$D2024-10-08T12:20:04.061Z","view_time":null,"first_public_time":"$D2023-10-11T12:57:38.429Z","fork":false,"stars":0,"starred":false,"owner":{"type":"team","id":"9b6646ec24dff9ba","login":"d-beilinson","name":"Daniil Beilinson","avatar_url":"https://avatars-next.observableusercontent.com/2d93226084d84c384a572996c4ef629251636290e2358a9491c05b7a329cc05b","tier":"starter_2024"},"version":448,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"ebe552f4a6dc3f7f","login":"daniil-beilinson","name":"Daniil Beilinson","avatar_url":"https://avatars-next.observableusercontent.com/b2df9a708eacdc151433d21ffde2c2a5a7e14023c8175bb08d2f4f6aceb3b7ce","tier":null}],"members":[{"type":"individual","id":"ebe552f4a6dc3f7f","login":"daniil-beilinson","name":"Daniil Beilinson","avatar_url":"https://avatars-next.observableusercontent.com/b2df9a708eacdc151433d21ffde2c2a5a7e14023c8175bb08d2f4f6aceb3b7ce","tier":null,"role":"editor"},{"type":"team","id":"9b6646ec24dff9ba","login":"d-beilinson","name":"Daniil Beilinson","avatar_url":"https://avatars-next.observableusercontent.com/2d93226084d84c384a572996c4ef629251636290e2358a9491c05b7a329cc05b","tier":"starter_2024","role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"title":"Repression data API","cells":[{"id":0,"value":"# Repression data API","pinned":false,"hidden":false,"mode":"md"},{"id":3,"value":"This document describes OVD-Info API providing the data about criminal political persecutions in Russia.","pinned":false,"hidden":false,"mode":"md"},{"id":33,"value":"The dataset contains detail information about the cases which took place since 2011. The data is used to power some of our projects, like the database of repressions for public anti-war stance available on https://repression.info.\nThe data published through this API contains more data and might be useful for researches and journalists.","pinned":false,"hidden":false,"mode":"md"},{"id":36,"value":"Besides the API this data available in shortened form [in CSV format](https://api.repression.info/v1/data?subset=full&csv=true).","pinned":false,"hidden":false,"mode":"md"},{"id":7,"value":"## Schema","pinned":false,"hidden":false,"mode":"md"},{"id":84,"value":"The data generally available in two formats:\n1. Flat data, e.g. in single collection of records\n2. Objective data, e.g. collections of 10 entities interconnected with each other.","pinned":false,"hidden":false,"mode":"md"},{"id":86,"value":"While working with flat data is easier and sometimes more useful, the original source of data is non-flat due to its nature and objective data contains more information. \n\nThe data available in 2 _languages_: Russian and English. In order to reduce the size of transmitted data, all the fields included in different _subsets_. _Basic_ subset contains only basic attributes, whereas _detailed_ and _full_ containing more data. \n\nThe schema can be requested by GET query using this URL: `https://api.repression.info/v1/schema`. \nAvailable options through url query params:\n1. type (`type=flat` will return flat schema)\n2. language (`ru` or `en`)\n \nAs a result, you will get a JSON object:","pinned":false,"hidden":false,"mode":"md"},{"id":101,"value":"getData('schema?type=flat')","pinned":false,"hidden":false,"mode":"ojs"},{"id":92,"value":"Below you may find detail description of all models. ","pinned":false,"hidden":false,"mode":"md"},{"id":72,"value":"viewof schemaLanguage = Inputs.radio([{ label: \"Russian\", value: \"ru\" }, { label: \"English\", value: \"en\" }], { label: \"Select schema language\", value: \"en\", valueof: v => v.value, format: v => v.label })","pinned":false,"hidden":false,"mode":"ojs"},{"id":82,"value":"### Flat data","pinned":false,"hidden":false,"mode":"md"},{"id":56,"value":"\n \n \n \n \n \n \n \n \n \n ${Object.values(await getData('schema?type=flat&language=' + schemaLanguage)).map(field => htl.html`\n \n \n \n \n `)}\n \n

Field ID	Field data type	Subset	Description
${field.id}	${field.dataType.length > 1 ? 'array of ' + field.dataType[0].replace('reference', 'string') + 's' : field.dataType[0].replace('reference', 'string')}	${field.subset.join(', ')}	${schemaLanguage === 'ru' ? field.descriptionRu : field.descriptionEn}

","pinned":false,"hidden":false,"mode":"html"},{"id":107,"value":"### Objective data","pinned":false,"hidden":false,"mode":"md"},{"id":417,"value":"$2f","pinned":false,"hidden":false,"mode":"ojs"},{"id":110,"value":"{\n const schema = await getData('schema?language=' + schemaLanguage)\n const els = Object.keys(schema).map(entityType => htl.html`\n

${entityType}

\n \n \n \n \n \n \n \n \n \n \n ${Object.values(schema[entityType]).map(field => htl.html`\n \n \n \n \n `)}\n \n

Field ID	Field data type	Subset	Description
${field.id}	${field.dataType.length > 1 ? 'array of ' + field.dataType[0].replace('backreference', 'reference') + 's' : field.dataType[0].replace('backreference', 'reference')}	${field.subset.join(', ')}	${schemaLanguage === 'ru' ? field.descriptionRu : field.descriptionEn}

\n `)\n return html`${els}`\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":8,"value":"## Rest API","pinned":false,"hidden":false,"mode":"md"},{"id":18,"value":"### Flat data","pinned":false,"hidden":false,"mode":"md"},{"id":180,"value":"$30","pinned":false,"hidden":false,"mode":"md"},{"id":131,"value":"flatData = getData(`data?preset=antiwar&language=${flatDataLanguage}&subset=${flatDataSubset}`)","pinned":false,"hidden":false,"mode":"ojs"},{"id":192,"value":"To see how the data will change based on parameters, look at the table below with results and change the options.","pinned":false,"hidden":false,"mode":"md"},{"id":146,"value":"viewof flatDataLanguage = Inputs.radio([{ label: \"Russian\", value: \"ru\" }, { label: \"English\", value: \"en\" }], { label: \"Flat data language\", value: \"en\", valueof: v => v.value, format: v => v.label })","pinned":false,"hidden":false,"mode":"ojs"},{"id":150,"value":"viewof flatDataSubset = Inputs.radio([{ label: \"Basic\", value: \"basic\" }, { label: \"Detailed\", value: \"detailed\" }, { label: \"Full\", value: \"full\" }], { label: \"Flat data options\", value: \"basic\", valueof: v => v.value, format: v => v.label })","pinned":false,"hidden":false,"mode":"ojs"},{"id":134,"value":"Inputs.table(flatData.data)","pinned":false,"hidden":false,"mode":"ojs"},{"id":196,"value":"Requested data can be easily used to perform calculations or render graphs. For example, below, you can see how this data used to quickly render the plot showing the current status of people persecuted for anti-war stance and gender proportion for each status. ","pinned":false,"hidden":false,"mode":"md"},{"id":160,"value":"Plot.plot({\n marginLeft: 300,\n y: { label: \"Person state\" },\n x: { label: \"Quantity\" },\n color: {scheme: \"Paired\"},\n marks: [\n Plot.barX((await getData('data?preset=antiwar&language=en')).data, Plot.groupY(\n { x: \"count\" },\n { fill: \"gender_en\", y: \"persecution_location_en\", tip: true, sort: { y: \"x\", reverse: true } }\n ))\n ]\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":23,"value":"### Objective data","pinned":false,"hidden":false,"mode":"md"},{"id":279,"value":"The objective data API presents more complex data for 10 available entities: _${Object.keys(await getData('schema')).join(', ')}_ which schema presented above and correspondent collections.","pinned":false,"hidden":false,"mode":"md"},{"id":316,"value":"\n \n \n \n \n \n \n \n \n ${Object.keys(await getData('schema')).map(entityType => htl.html`\n \n \n \n `)}\n \n

Entity name	Collection name	Collection url
${entityType}	${entityType !== 'city' ? entityType + 's' : 'cities'}	${API_ENDPOINT}/${entityType !== 'city' ? entityType + 's' : 'cities'}

","pinned":false,"hidden":false,"mode":"html"},{"id":277,"value":"#### Listing records","pinned":false,"hidden":false,"mode":"md"},{"id":208,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":222,"value":"await getData(`persons?preset=antiwar&subset=full`)","pinned":false,"hidden":false,"mode":"ojs"},{"id":371,"value":"Objective data can be easily used to perform calculations on entity relations and render graphs. For example, below, you can see how this API used to quickly render the plot showing most popular criminal articles used in political persecutions of women.","pinned":false,"hidden":false,"mode":"md"},{"id":373,"value":"Plot.plot({\n marginLeft: 70,\n y: { label: \"Article of criminal code\", type: \"band\" },\n x: { label: \"Quantity\" },\n color: {scheme: \"Accent\"},\n marks: [\n Plot.barX((await getData('persons?language=en&fields=[\"person_articles\",\"article_number\"]&filter={\"person_gender_en\":\"female\"}')).data.map(rec => rec.attributes.person_articles.map(article => article.article_number)).flat(), Plot.groupY(\n { x: \"count\", filter: v => v.length > 9 },\n { fill: \"article_number\", tip: true, sort: { y: \"x\", reverse: true } }\n ))\n ]\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":273,"value":"#### Obtaining entity data","pinned":false,"hidden":false,"mode":"md"},{"id":219,"value":"$32","pinned":false,"hidden":false,"mode":"md"},{"id":263,"value":"viewof objectDataLanguage = Inputs.radio([{ label: \"Russian\", value: \"ru\" }, { label: \"English\", value: \"en\" }], { label: \"Language options\", value: \"en\", valueof: v => v.value, format: v => v.label })","pinned":false,"hidden":false,"mode":"ojs"},{"id":236,"value":"viewof objectDataSubset = Inputs.radio([{ label: \"Basic\", value: \"basic\" }, { label: \"Detailed\", value: \"detailed\" }, { label: \"Full\", value: \"full\" }], { label: \"Subset options\", value: \"basic\", valueof: v => v.value, format: v => v.label })","pinned":false,"hidden":false,"mode":"ojs"},{"id":256,"value":"viewof objectDataReferences = Inputs.radio([{ label: \"Title only\", value: \"title\" }, { label: \"Basic\", value: \"basic\" }, { label: \"Detailed\", value: \"detailed\" }, { label: \"Full\", value: \"full\" }], { label: \"Reference options\", value: \"basic\", valueof: v => v.value, format: v => v.label })","pinned":false,"hidden":false,"mode":"ojs"},{"id":231,"value":"JSON.stringify(await getData(`persecutions/recyu0H7heTQjuOTM?language=${objectDataLanguage}&subset=${objectDataSubset}&references=${objectDataReferences}`), null, 4)","pinned":false,"hidden":false,"mode":"ojs"},{"id":283,"value":"#### Listing entity references","pinned":false,"hidden":false,"mode":"md"},{"id":287,"value":"The general data on references of particular entity can be requested by GET query using this URL: https://api.repression.info/v1/[entityCollection]/[entityId]/references. \nThe query takes the same parameters as general collection listing. \nListing of entities of specific type could be requested by this URL: https://api.repression.info/v1/[entityCollection]/[entityId]/references/[referencedEntityCollection]. \n\nTake a look how the data will change for listing of references Alexey Gorinov persecution depending on reference type values:","pinned":false,"hidden":false,"mode":"md"},{"id":350,"value":"viewof referenceCollection = Inputs.select([...new Set(\n [''].concat((await getData('persecutions/recyu0H7heTQjuOTM/references')).data.map(ref => ref.type === 'city' ? 'cities' : ref.type + 's'))\n)], {label: \"Reference of specific type\"})","pinned":false,"hidden":false,"mode":"ojs"},{"id":345,"value":"JSON.stringify(await getData(`persecutions/recyu0H7heTQjuOTM/references${referenceCollection === '' ? '' : '/' + referenceCollection}?language=en&subset=basic&references=basic`), null, 4)","pinned":false,"hidden":false,"mode":"ojs"},{"id":427,"value":"## Change log","pinned":false,"hidden":false,"mode":"md"},{"id":441,"value":"### 2024-10-08","pinned":false,"hidden":false,"mode":"md"},{"id":445,"value":"- New fields for entity _person_ of objective API: `person_imprisoned` (_true_ if person imprisoned now), `person_persecuted` (_true_ if person under persecution now)\n- New fields for entity _persecution_ of objective API and for flat API: `persecution_imprisoned` (_true_ if person imprisoned now), `persecution_actual` (_true_ if person under persecution now)\n- New presets: _actual_ will filter records related to actual persecutions, _imprisoned_ will filter records with imprisoned people, _after-invasion_ will filter records related to persecution started on 24/02/2022 or afterwards, _before-invasion_ will filter records related to persecution started before 24/02/2022","pinned":false,"hidden":false,"mode":"md"},{"id":429,"value":"### 2024-01-16","pinned":false,"hidden":false,"mode":"md"},{"id":431,"value":"- New fields with unique IDs of records: `persecution_id` (flat and objective data)`, person_id, criminal_case_id, sentence_id, restraint_measure_id, imprisonment_id, article_id, location_id, protest_event_id, city_id, evaluation_id`\n- New fields with time of last change: `last_updated` (flat data)`, person_last_updated, persecution_last_updated, criminal_case_last_updated`","pinned":false,"hidden":false,"mode":"md"},{"id":10,"value":"## Client library","pinned":false,"hidden":false,"mode":"md"},{"id":254,"value":"TBD","pinned":false,"hidden":false,"mode":"md"},{"id":41,"value":"## Helpers","pinned":false,"hidden":false,"mode":"md"},{"id":44,"value":"API_ENDPOINT = 'https://api.repression.info/v1'","pinned":false,"hidden":false,"mode":"ojs"},{"id":48,"value":"getData = method => fetch(`${API_ENDPOINT}/${method}`)\n .then(response => {\n if (!response.ok) throw new Error(response.status)\n return response.json();\n })","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W33","stdlib":"1","titling":"auto"}} 2d:"$2c:body" 28:E{"digest":"NEXT_REDIRECT;replace;/@d-beilinson/repression-data-api;307;"}