Details of the EDIT-function Inverted Index structure

How to extract data from the inverted index structure.

• InvIdx is a character string concatenation of 6 segments, separated by linefeeds ($LF):

  	meaning	format	from	to
  1 PREFIX	values for FMT, LEX, LEX1, IDs1, POSspW1, IDsinW1, POSs1, nWORDs, nIDs, nPOSs	Text	1	LEX1 - 1
  2 LEXICON	alfabetical list of indexed words	Text	LEX1	IDs1 - 1
  3 IDs	list of documents indexed	Text	IDs1	POSspW1 - 1
  4 POSpW	for each word: count + positions	int4	POSspW1	IDsinW1 - 1
  5 IDsinW	for each word: 1 bit per ID int4_count= CEILING(nIDs/32)	int4	IDsinW1	POSs1 - 1
  6 POSs	sequential for each ID: count, positions in document	int4	POSs1	LEN_TRIM(InvIdx)

• To extract data from InvIdx:
  • CHARACTER FMT*4, LEX*6
  • DIMENSION vec(1E6)
  • INIT(InvIdx, FMT, LEX, LEX1,IDS1,POSSPW1,IDSINW1,POSS1,NWORDS,NIDS,NPOSS)
  • ALIAS(InvIdx,1, PREFIX,lex1-2, D1,1, lexicon,ids1-lex1-1, LF1,1, IDs,POSSPW1-ids1, POSpW,IDSINW1-POSSPW1, IDsinW,POSs1-IDSINW1, POSs,4*nPOSs)
  • ! LF1 is the separating line feed character. Only useful to display a lexicon list box
  • IF(FMT(1:3) == "hex" ) FMT = "(Z" // FMT(4) // ")"
  • IF(FMT(1:3) == "int" ) FMT = "(ib4)"
  • ALIAS(vec,1, POSsPW_vec,nWords, IDsinW_vec, NWORDS, POSs_vec,nPOSs)
  • READ(Text=POSpW, Format=FMT) POSsPW_vec
  • READ(Text=IDsinW, Format=FMT) IDsinW_vec
  • READ(Text=POSs, Format=FMT) POSs_vec
• Example:
  • InvIdx = "FMT=hex3"
  • EDIT(T="It is, what it...is", ID="T1", II=InvIdx)
  • EDIT(T="What is it?", ID="T2", II=InvIdx)
  • EDIT(T="It is a banana !", ID="T3", II=InvIdx)

  PREFIX	FMT=hex3 LEX=UPDATE LEX1=129 IDS1=149 POSSPW1=158 IDSINW1=174 POSS1=190 NWORDS=5 NIDS=3 NPOSS=22	LEX1-1 characters default FMT=int4
  LEXICON	a banana is it what	5 Words, sorted alfanum, space separated
  IDs	T1.T2.T2.	3 IDs in order of call, . denote linefeed characters

  written with FMT	a	banana	is	it	what
  POSspW	2	2	7	7	4	occurences (includes count)
  IDsinW	4	4	1+2+4=7	1+2+4=7	1+2=3	Bitsum of ID numbers that have word.
  POSs	2 7	2 9	3 4 12 2 6 2 4	3 1 D 2 9 2 1	2 8 2 1	count and positions in ID, HEX

  • DLG(LB=lexicon, LB=IDs, E=POSsPW_vec,F="3F10", E=IDsinW_vec,Y, E=POSs_vec,Y, Ti=PREFIX)