Say Hello to GLAMpipe!

Documentation in progress...

GLAMpipe is an extensible open source app for browsing, transforming and uploading GLAM datasets.

The basic idea of GLAMpipe is to read data and files from different sources, process the metadata and export the data and files to different destinations. The workflow is based on documents and nodes instead of rows and cells.

Terminology

Projects

Projects groups collections and nodes.

Collections

A collection holds data objects and there can be several collections in one project. Technically speaking collection is a collection in MongoDB.

Nodes

Nodes are the building blocks of GLAMpipe data handling. The idea of a node is that each node makes one, relatively simple thing with the data or file. By chaining nodes one can make different transformations to the data.

Nodes are divided to four main categories:

  • Importing data
  • Processing the data
  • Exporting the data
  • VIewing the data

Nodes are mini-applications written in Javascript that have their own user interface. Javascript in nodes is not part of GLAMpipe's code base but it is executed in VM.

Scripts can not make any asynchronous operations like write files or make requests to external APIs directly (see Cores).

Cores

Cores can write files, extract text from PDFs, make web requests and so on. Scripts in GLAMpipe's nodes can pass parameters to cores which then return the result of operation back to node.