2b:null 2c:["$","title",null,{"children":"About Simple Free-Text Queries | James Anderson | Observable"}] 31:T1734,## Alternative Implementations The approach was chosen after considering the alternatives demonstrated by other implementations. ### w3c: https://www.w3.org/2009/sparql/wiki/Feature:FullText http://www.w3.org/TR/xpath-full-text-10/ They propose to add a `contains` operator for filters ?s ?p ?o . FILTER ( ?p IN (dct:title, foaf:name, rdfs:label)) . FILTER ( ?o contains 'foo*' ) . or FILTER ( ?o contains ('foo*', constraint) ) . they propose also to adopt xquery text matching syntax. The grammars are https://www.w3.org/TR/xpath-full-text-10/#id-grammar (@197 productions v/s 39 ?) https://www.w3.org/TR/xpath-full-text-10/#id-xpath-grammar (@125 productions v/s 70 https://www.w3.org/TR/xpath20/#nt-bnf) In order for this approach to improive on regular expression filter constraints, it would have to be compiled to generate index results generation rather than scan a filter. ### stardog : https://www.stardog.com/docs https://docs.stardog.com/query-stardog/full-text-search It supports both magic properties and a sub-query form. For magic properties, they follow Jena ```sparql SELECT DISTINCT ?s ?score WHERE { ?s ?p ?l. (?l ?score) ('mac' 100). } ``` in general (term score) (pattern limit) or (term score) (pattern minScore limit) The subquery/service clause form permits more arguments and results. SELECT * WHERE { service fts:textMatch { [] fts:query 'Mexico AND city' ; fts:threshold 0.6 ; fts:limit 10 ; fts:offset 5 ; fts:score ?score ; fts:result ?res ; } } ### jena : https://jena.apache.org/documentation/query/text-query.html says, 'the text index provides an inverted index that maps query string matches to subject URIs' but, in fact, the relation is (subject x graph x string x language) plus an identifier argument (to distinguish predicates ?) PREFIX ex: PREFIX rdfs: PREFIX text: SELECT ?s ?lbl WHERE { ?s a ex:Product ; rdfs:label ?lbl . ( ?s ?score ) text:query (rdfs:label 'printer') ; } This is not complete: the general intent is not to constrain just the text term, but to index the entires statement with an graph context. WHERE { ?s a ex:Product ; ( ?s ?score ?lbl) text:query (rdfs:label 'printer') ; } The general syntax permits literals in the subject position as well ?s text:query 'word' # query ?s text:query ('word' 10) # with limit on results ?s text:query (rdfs:label 'word') # query specific property if multiple ?s text:query (rdfs:label 'protégé' 'lang:fr') # restrict search to French (?s ?score) text:query 'word' # query capturing also the score (?s ?score ?literal) text:query 'word' # ... and original literal value (?s ?score ?literal ?g) text:query 'word' # ... and the graph where the form is further qualified by its appearance in a graph clause. (see https://jena.apache.org/documentation/query/text-query.html#queries-with-graphs) ### dotnet : https://github.com/dotnetrdf/dotnetrdf/wiki/UserGuide-Full-Text-Querying-With-SPARQL It follows ARQ PREFIX pf: SELECT * WHERE { ?match pf:textMatch ( "text" 0.75) . } ### marklogic : https://docs.marklogic.com/guide/semantics/semantic-searches#id_43103 SELECT * WHERE{ ?s ?p ?o . FILTER cts:contains(?o, cts:or-query(("Monarch", "Sovereign"))) FILTER(?p IN (dc:description, rdfs:type)) } ### neptune : https://docs.amazonaws.cn/en_us/neptune/latest/userguide/full-text-search-sparql-examples.html It follows the stardog pattern which expresses the freetext match as a service clause PREFIX foaf: PREFIX neptune-fts: SELECT * WHERE { SERVICE neptune-fts:search { neptune-fts:config neptune-fts:endpoint 'http://your-es-endpoint.com' . neptune-fts:config neptune-fts:queryType 'match' . neptune-fts:config neptune-fts:field foaf:name . neptune-fts:config neptune-fts:query 'michael' . neptune-fts:config neptune-fts:return ?res . } } ### virtuoso : http://docs.openlinksw.com/virtuoso/rdfsparqlrulefulltext/ It relies on `bif:contains` as a "magic predicate", to indicate that the respective object serve as a match pattern. Rather than incorporating additional parameters and results with standard syntax, it handles return properites with an `OPTION` clause PREFIX bif: SELECT * WHERE { ?s ?p ?o . ?o bif:contains 'NEW AND YORK' OPTION (score ?sc) . # this form fails } ORDER BY DESC (?sc) LIMIT 10 ### graphdb : http://graphdb.ontotext.com/documentation/free/full-text-search.html The free text capability is based on Lucene, but extends beyound elementary text 'node' matching to incorporate graph patterns (called 'molecules') into the index itself. These introduce additional bindings directly into the result solutions. Each index is designated by its own predicate. Additional parameters are indicated by specific predicates. A query result includes the subgraph implicated by matched "nodes". PREFIX luc: PREFIX ex: SELECT * { ?node luc:myIndex "lucene query string" . ?node ex:importance ?importance . ?node luc:score ?score . } ORDER BY ( ?score + ?importance ) 32:T1429,## Dydra Implementation ### desired capabilities - Provide the capability with as little configuration as possible. This includes the operations to - create the index with an administrative `POST` request to the account's adminstrative resource. - update it explicitly with a `DELETE` request to its content resource. - delete it explicitly with a `DELETE` request to its adminstrative resource. - Make graph qualification a consequence of the statement pattern context only. Each index contains text from one repository only, but text index itself is context-free. - The service forms (fts:textMatch in StarDog and neptune-fts:search in Neptune) are not necessary. They are syntactic reformulations of subselects, which the compiler can construct from the the equivalent appearance of a distinguished predicate. The only issue is the parameter syntax. - The match expressions should avoid redundant elements. The appearance of a term should either indicate an argument or a result binding. In a form like ?s text:query (rdfs:label 'printer'); rdfs:label ?lbl . is to be avoided as the `rdfs:label` indicates both a match constraint and a statement pattern constraint. - Although formulations as filters appear simpler, they mislead. Filtered terms customarily consume results from ongoing solutions. This makes it improper to place variables in a filter expression to bind the secondary results. They are more appropriate in a subject list. - The object list can include constraints as positional arguments. If their number increases and the precedence is not obvious, then a property list path would be appropriate. - The indicator should be a special predicate - any one of those known. - The parameters on the right side should be - the respective object term - additional constraints; either a single argument for a limit or a property list for others - It would be easy to create a text index upon first reference, but that introduces the possibilities of both inordinate resource consumption and inadvertent reference errors. - Create the index by posting to the account - Update the index content by clearing the index repository. This is accomplished with a graph store `DELETE`request. An alternative would be to use the indicator predicate to designate a graph in the source repository. - Delete the index by deleting its adminstrative resource. - The subject list binds the results - the matched string - the score attribute ### This yields the alternatives: #### alternative 1.1 : magic property binding subject ?s a ex:Product ; rdfs:label ?lbl . ( ?s ?score ) text:query (rdfs:label 'printer' 'lang:x') . filter( ?score > 1 ) or ?s a ex:Product ; rdfs:label ?lbl . ( ?s ?score ?lbl ) text:query (rdfs:label 'printer' 'lang:x') . filter( ?score > 1 ) #### alternative 1.2 : magic property binding literal term ?s a ex:Product ; rdfs:label ?lbl . (?lbl ?score) (pattern limit) #### alternative 2 : service clause (see above) #### alternative 3 : filter clause ?s a ex:Product ; rdfs:label ?lbl . filter ( text:query(?lbl 'printer' 'lang:x') # no where to put returned properties : ( ?s ?score ) Of which - the service clause is unnecessarily complex, - alternative 1.1 means the subject is redundantly indexed, - alternative 3 cannot bind search result properties. Which leaves alternative 1.2 ### and an implementation architecture The implementation involves two components: - A facility which accepts texts and constructs a free text index for the strings in possible relation to materialized content. - A means within the SPARQL processor to incorporate those indexes in the query evaluation process. The index facility relies on [PostgreSQL](https://www.postgresql.org/docs/12/textsearch.html) free text support. The interpretation process involves three steps: - During BGP macroexpansion, the indicator predicate causes the subject and object argument list be be coalesced into a function invocation predicate term, to leave just that one stagement to be interpreted. - Adopt `` as the standard, but with the variant signature rearrangements to permit the BGP translation to isolate the function calls. - Allow for ```sparql ?lbl pattern (?lbl) (pattern) (?lbl) (pattern limit) (?lbl ?score) (pattern) (?lbl ?score) (pattern limit) ``` - The compiled BGP invokes the match function with any in-line bindings. - The match implementation initiates the match and passes any results through in the same manner as a scanned statement pattern. If there is a score filter, it is interposed before any result is passsed onwards in the evaluation process. See the PostgreSQL [documentation](https://www.postgresql.org/docs/9.6/textsearch-controls.html) for an explanation of the ts_rank score, 33:T10f7,### SPARQL inline reference ``` curl_sparql_request -X POST --data-binary @- \ -H "Content-Type: application/sparql-query" \ -H "Accept: application/sparql-results+json" \ --account "sba" \ --repository "ppp" \ '$pattern=%22ZWOLLE:*%22' < where { # alternative with full subject description # ?subject ?p ?o . # bind (\$pattern as ?CITY) ?subject ?city . # alternative with freetext parameter ?city \$pattern . # alternative with constant filter # filter (?city = "ZWOLLE") # alternative with parameterized filter # filter (?city = \$pattern) } EOF ``` The variant with the complete subject description relies on a full repository scan, which takes about six seconds for the sixty-six statement `sba/ppp` repository. The free text formulation which retrieves just the subjects responds in approximately 0.6 seconds. This is twice the time of a variant which filters against a constant, as that is pushed into the BGP, but less than the 1.6 seconds for the case where the filter acts against a parameter. The subject-only formulation yields: ``` { "head": { "vars": [ "subject" ] }, "results": { "bindings": [ { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_2685"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_3664"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_9134"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_9574"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_10707"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_10824"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_13201"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_14835"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_15323"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_17321"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_18388"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_20369"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_20622"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_21218"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_26687"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_27402"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_27499"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_31081"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_32324"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_34372"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_36034"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_37881"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_41344"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_41577"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_41798"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_42704"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_42705"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_43434"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_44319"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_44747"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_46028"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_48083"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_49632"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_53784"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_54387"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_57858"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/Louisiana_63434"} }, { "subject": {"type":"uri", "value":"http://nl4.dydra.com/150k_plus_274897"} } ] } } ``` 34:[] 2f:{"id":"cae13bb44f1588b9","slug":"about-free-text-queries","title":"About Simple Free-Text Queries","thumbnail":"93dd35ef1eae710788276fc4a15fb265d0dfbb9d0ee4764d648cc252314174b2","access_level":"public","create_time":"$D2021-07-03T22:59:47.221Z","update_time":"$D2024-08-10T12:24:13.941Z","view_time":null,"first_public_time":"$D2021-07-03T23:59:50.391Z","fork":{"id":"7f9629bf197ebf02","slug":"about-rdf-annotation","title":"About RDF annotation","owner":{"id":"b452fd69977f5701","login":"lomoramic","name":"James Anderson"}},"stars":1,"starred":false,"owner":{"type":"team","id":"b452fd69977f5701","login":"lomoramic","name":"James Anderson","avatar_url":"https://avatars-next.observableusercontent.com/dc520bbc733259f2913522cf1d50607777b98a3083f290a6f182534561896497","tier":"starter_2024"},"version":789,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"987c90ad70c449c1","login":"lomoramic","name":"James Anderson","avatar_url":"https://avatars-next.observableusercontent.com/dc520bbc733259f2913522cf1d50607777b98a3083f290a6f182534561896497","tier":null}],"members":[{"type":"individual","id":"987c90ad70c449c1","login":"lomoramic","name":"James Anderson","avatar_url":"https://avatars-next.observableusercontent.com/dc520bbc733259f2913522cf1d50607777b98a3083f290a6f182534561896497","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"subscribed":null,"forks":3,"collection":null,"type":"notebook","body":{"title":"About Simple Free-Text Queries","cells":[{"id":611,"value":"md`# About Simple Free-Text Queries\n## Support for free-text in Dydra\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":0,"value":"md`\n\nDydra provides two text index forms.\nOne approach recognizes an active predicate (a magic property) as an operator in a BGP.\nThe other supports free text arguments to parameterized SPARQL view requests.\nThe BGP variant accepts the pattern plus optional constraints and yields matched results to the BGP evaluation process.\nThe SPARQL view variant accepts arguments for pattern parameters to constrain the solutions returned from the materialized cache.\n\nBoth rely on PostgreSQL full text index support to extend materialized views.\n\nThis document describes the simple text index forms.\nThis mechanism matches patterns to literal strings and their rank.\nThe alternative, the [materialized text view](about-materialized-text-views), combines the text index with the additional terms projected for a materialized view.\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined"},{"id":595,"value":"$31","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":599,"value":"$32","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":711,"value":"## An Example\n\nThe [`sba/ppp`](https://dydra.com/sba/ppp) repository contains a version of the U.S. Small Business Adminstration's \"[Payroll Protection Program](https://www.sba.gov/funding-programs/loans/covid-19-relief-options/paycheck-protection-program)\" disbursement records.\nThey amount to over sixty-six million statements and 1.3 million string terms.\n","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined"},{"id":706,"value":"### creation\nThe following request creates a simple text index\n```sh\ncurl -X POST -s -w \"%{http_code}\\n\" -u \":\\${STORE_TOKEN}\" \\\n -H \"Accept: application/sparql-results+json\" \\\n -H \"Content-Type: application/json\" \\\n --data-binary @- \\\n \"\\${STORE_URL}/system/accounts/sba/repositories\" <