InvertedIndexQuery: Query an existing Inverted Index

A left hand side argument to HicEsts EDIT function with the keyword InvertedIndex=existing_index will be assigned the queried text or numeric result.

• • RESULT = EDIT(Text=QUERY, [ options, ] InvertedIndex=InvIdx)
  inverted_index_keywords
  useful in a query:

  keyword	position	short	action
  Text	first, required	T=QUERY	string containing the words to be analyzed.
  IDentification	middle optional	ID=number ID=string	only document number only documents with IDs containing string (Regex expressions possible) found = EDIT(T="father | mother son | daughter, ID="k..g", ii=InvIdx) ! find all (father OR mother) AND (son OR daughter) in all IDs containing e.g. "king" or "kong"
  Option	middle optional	opt = 128	128 for a RegEx QUERY 32 for an "almost" QUERY 1 for a case sensitive QUERY (only if InvIdx was built with this option)
  SorTSequence	middle optional	sts=23	sort_query_result (23 means: first sort column2, then subsort column3). Default is 32. No-sort is 0.
  EXTRA	middle optional	extra=xtra	query_direction xtra="D=R" works off query string left to right (also possible "r", "F", "f", "+"). "D=L" works right to Left. left_expand result to an EXTRA-initialized character x by xtra="L=x" right_expand by xtra="R=x" exclude_occurence of x from result by "X=x". The xtra partial strings can be combined, e.g. if invidx was set up for indexing linefeeds and equals by: EDIT(Text=original, EXTRA="="//$LF, InvertedIndex=invidx) result = EDIT(Text="alfa=", EXTRA="D=R X=x L=x R=x", InvertedIndex=invidx) ! x == $LF will find all occurences of alfa followed by = with no linefeed between alfa and = expanded to the next linefeed left of alfa and to the next linefeed right of =.
  InvertedIndex	last, required	ii=invidx	the inverted index string to be used for this QUERY.

• result_type
  scalar, array, string, or vector determines RESULT:

  Find = "Question: to be to" in Shakespeare Complete Works	RESULT
  scalar = EDIT(T=find, II=InvIdx)	n = 156 occurences of least frequent word. This is the upper limit of find-string occurences.
  array = EDIT(T=find, II=InvIdx) string = EDIT(T=find, II=InvIdx)	fills first 145 rows or maximum allocated length column explain 1 start location of result in original 2 spread (character count between start and end location) 3 relevance (> 0 number of occurence in ID, <0 with Option 32: missing elements of OR/RegEx groups) 4 array: start of ID string in InvIdx. End is linefeed $LF string: document ID Use option SorTSequence to change default sort.
  vector = EDIT(T=find, II=InvIdx)	word counts and locations in POSs ⇾ see details. Element 1 = ID1-QUERY-word1 count position Element 2 = ID1-QUERY-word2 count position and so on for all IDs and QUERY words

• standard_query
  examples (Option not set or zero):

  QUERY string	QUERY for xx, yy, zz, etc full word(s)
  xx yy zz	xx AND yy AND zz must be present
  xx | yy | zz	xx OR yy OR zz
  xx | yy zz	(xx OR yy) AND zz
  xx |yy 1|2|3	(xx OR yy) AND (1 OR 2 OR 3)
  <10 xx yy zz	10 is the maximum spread of RESULT
  xx <1L zz	maximum 0 characters Left (no back steps). Use "-", "b", "B", "l" instead of "L" if you like.
  xx <5R zz"	maximum 4 characters to next word (limit forward QUERY). Use "+", "f", "F", "r" instead of "R" if you like.
  <100 xx <4- <12+ yy|zz pp|qq	RESULT max 100 chars wide, not more than 3 chars back, not more than 11 chars foreward find all occurences of xx, then yy or zz, then pp or qq.

• near_query
  with Option=32 will also show partial RESULTs for multi-OR-groups searches with at least 1 completely missing OR group.
• regular_expression_query
  with Option=128 (RegEx), ⇾ RegEx:

  QUERY string	sample RESULTs found in Shakespeare Complete Works
  m?other	mother,other
  b.a.e	blade,blame,blaze,brace,brake,brave
  gr.*t	great,greatest,greet,grievest,groat,grumblest,grunt
  b.a.e gr.*t	great part of blame brave Master Shootie the great

• • A simple query for Hamlets "The to-be-to question." is shown, sorted along column 2. In a standard query the sequence of search words is irrelevant:
    • DIMENSION array(1000,4) ! to get a maximum of 1000 hits
    • array = EDIT(Text="The to-be-to question.", SorTSequence=2, InvertedIndex=InvIdx)
    • hits = LEN_TRIM(array) / 4 ! = 145 non-zero rows in array

    column 1 hit position in ALL (only 2 rows shown)	column 2 spread of result	column 3 hit nr in ID	column 4 start of ID-string in InvIdx
    1124245	41	30	193660
    156590	83	6	193660 etc.

  • One way to visualize the result:
    • found = " "
    • DO i = 1, 2 ! 2 of 145 rows, from line begin to clip end. Many other clippings are possible.
    •     EDIT(T=ALL, SetPosition=array(i,1), Left=$CR, Mark1, SetPosition=array(i,1)+array(i,2), Mark2, APpendTo=found)
    • ENDDO

    Only the first 2 rows of found (out of 145) are shown with SorTSequence=2: smallest spread first

    Ham. To be, or not to be- that is the question:

    COUNTESS. To be young again, if we could, I will be a fool in question, hoping to be the

  • The LeXicon keyword can serve to highlite the hits:
    • found = EDIT(Text=found, Option=2, $Marks="[]", LeXicon="The to-be-to question.")

    If the marks used ("[]" ) are unique in ORIGINAL, they can be further expanded to generate a variety of secondary information like sophisticated table of contents etc.

    Ham. [to] [be], or not [to] [be]- that is [The] [question]:

    COUNTESS. [to] [be] young again, if we could, I will [be] a fool in [question], hoping [to] [be] [The]

• ⇾ Inverted index data structure details