Skip to content

Scaling up

No due date 50% complete

Geneve works similarly to a language compiler. Briefly:

  1. The data specs are parsed and translated into abstract syntax trees (AST)
  2. The ASTs are traversed and the fields constraints are collected in an intermediate representation (IR)
  3. The constraints of each field are solved, if possible, and stored in sets of field-value pairs
  4. The sets of field-value pair…

Geneve works similarly to a language compiler. Briefly:

  1. The data specs are parsed and translated into abstract syntax trees (AST)
  2. The ASTs are traversed and the fields constraints are collected in an intermediate representation (IR)
  3. The constraints of each field are solved, if possible, and stored in sets of field-value pairs
  4. The sets of field-value pairs are translated to one or more documents
  5. The generated documents are ingested into the Stack

Whereas steps 1-2 are executed once, steps 3-5 are iterative and present a ceiling that is hit depending on:

  • number and complexity of the relations that generated data needs to satisfy (step 1 and 2)
  • number and complexity of the constraints problems to be solved for generating the documents (step 3)
  • number of documents to be generated (iteration of steps 3-5)

This milestone seeks solutions to the scalability limits due to having all the five steps above implemented in Python.