Build pipeline

ProcessSpace Specifications

concordance build chains its steps in a fixed order and produces one build. Each step is a pure function of the outputs of the previous ones and reports its anomalies as findings; no content anomaly stops the chain. Two builds on unchanged sources produce byte-identical files.

Steps

  1. Validate the configuration, read and validate the lock file it names, merge the profile and load the declared plugins; stop on a configuration error, a missing or invalid lock file, an invalid profile or a plugin that cannot be loaded.
  2. Ingest every source with cache and without writing anything into the repositories: a git source is cloned with its whole history and without the blobs, so that the checkout fetches only the files it reads; record for each file the last commit that touched it and its date, from the repository holding a local folder too when the file is unchanged since its commit.
  3. Parse the markdown and frontmatter into sections, lists, links and the text units the recognition reads.
  4. Read every other file a reader or a converter of the plugins accepts, in parallel: the reader gives its metadata and, for a transcript, its cues with their timecodes; the converter gives its PDF, cached by the fingerprint of the file, and the text of every page of that PDF, the only text an office document ever contributes. A conversion that fails is a finding and the document stays a download.
  5. Read the pseudonymisation dictionary the configuration names and validate it; report a missing or malformed file, as a warning while pseudonymisation is disabled, as an error that fails the build and withholds every transcript when it is enabled. Pseudonymise every transcript before typing reads it: the speakers and the text of its cues, the metadata of its reader, and the file offered for download, written back by the reader with the same timecodes; withhold a transcript whose reader cannot write it back, and every transcript unless the configuration publishes them; report the name-shaped mentions the dictionary does not cover.
  6. Type every note and every document through the cascade, compute identifiers (a document keeps its extension), resolve application and domain; produce an entity per note and per document, the metadata of a document as its attributes.
  7. Pseudonymise the notes and the documents of the entities whose type is in the configured scope, the meetings by default, with the same dictionary: rewrite the note and parse it again, replace the pages and properties of the document, and the title, aliases, summary and attributes of the entity; the rewritten note is what the fragments render.
  8. Run the source plugins on the typed entities: the contract importers read the contract each API note declares and add its endpoints, their exposes links, the candidate objects and one contract record per contract.
  9. Reconcile the notes and documents that look like twin resources, by declaration, base name, the title a reader reports against the heading of a note, or their texts compared without the stopwords of the locale: merge the groups above the merge threshold into one entity with several representations, the note's identifier when it has one, report the pairs above the candidate threshold; a pair the lock file merges is one entity whatever its score, grouped by the criterion lock file, a pair it separates is neither merged nor reported; report every document left without a markdown representation. The reconciliation comes before the dictionary, so that a twin folded into its note never counts as a homonym of that note.
  10. Build one recognition dictionary per locale from the titles and aliases of the entities, those of a folded twin naming the entity it became, glossary sources first, stopwords excluded; flag the homonyms.
  11. Scan every note and every page of every document with the dictionary of its locale and record each occurrence with its line, position, section and context; in a document the line is the number of the page, slide or cue and the section its label, so that a mention cites slide 3 or a timecode.
  12. Produce the links: written links, frontmatter references, mentions in mapped sections and in prose, co-occurrences per paragraph with the bounded neighbourhood; the files of a folded twin declare and receive links under its identifier, moved to the merged entity once the relations are typed.
  13. Combine the confidences of every link into one link per source, target, relation and attributes, every provenance kept.
  14. Type the relations: keep a declared relation the profile allows between the two types and drop the others, name a related link from its type pair or leave it related, capped and reported unless co-occurrence alone knows it.
  15. Discover the recurring expressions without a note in the notes and in the pages of the documents, headings and section labels left out as titles rather than usage, flag the undefined terms and publish the keyword pages above the threshold as entities; the rejected terms of the lock file, compared on their normalised form, and the real names of the pseudonymisation dictionary are never proposed.
  16. File under the domains the lock file promotes the notes it names among those no frontmatter, folder or glob files, with the origin lock; then, when the configuration asks for the emergent domains, walk the graph of the typed links and the co-occurrence neighbours from every term with a note whose degree reaches the threshold, attach every unclassified note within the radius to its closest pivot, report each one as a suggested domain and file it under the domain named after its pivot, with the origin inferred, only when the configuration assigns; a note with a declared domain is never touched.
  17. Run every check of the registry on the model, enrich the findings of every step with the same registry and sort them once.
  18. Write the model and the build log, the decisions of the lock file counted in its summary and the suggested domains listed with their pivots, then one fragment per entity next to them: the note rendered to sanitised HTML section by section, its written links turned into page links, its plain text and the text of every page of its documents for the search index, the original file and the PDF of each document kept for the page, a pseudonymised transcript written from its rewritten bytes rather than copied, or the passages of a keyword page. Nothing after this step reads a source.
  19. Resolve the theme: the components the plugins contribute and the project's theme.yaml, whose labels override the message catalogue of the project locale.
  20. Render the site from the model, the fragments, the profile and the theme, as concordance render does alone: the home page, one page per entity and per keyword page at the address of its identifier, the document viewer on the page of a document with its files placed next to it, the alphabetical index, the to-do page, the search index and the assets, every link relative to its page so that the site works over file://.
  21. Measure every page against the 150 kB budget and run the static accessibility checks; print the pages written, the size of each island bundle, the largest page and the findings, as warnings that never fail the build.

Properties5

Application
Concordance command line
Domain
Ingestion
Status
valid
Execution
service
Triggers
pipeline run, manual command

5 declared keys. The rest of the file is free text.

On this page1

  1. Steps
See the neighbourhood map6 pages6Neighbourhood mapBuild pipeline

Neighbourhood map Build pipeline

Distance1 hop

Types
existing pageword without a note

163 neighbours in total, more than the map shows.

The 6 neighbourstextual equivalent

Six neighbours at most, always named. Beyond that the map teaches nothing: the list takes over.