GLAMpipe is an extensible open source app for browsing, transforming and uploading GLAM datasets.
The basic idea of GLAMpipe is to read data and files from different sources, process the metadata and export the data and files to different destinations. The workflow is based on documents and nodes instead of rows and cells.
Projects groups collections and nodes.
A collection holds data objects and there can be several collections in one project. Technically speaking collection is a collection in MongoDB.
Nodes are the building blocks of GLAMpipe data handling. The idea of a node is that each node makes one, relatively simple thing with the data or file. By chaining nodes one can make different transformations to the data.
Nodes are divided to four main categories:
Scripts can not make any asynchronous operations like write files or make requests to external APIs directly (see Cores).
Cores can write files, extract text from PDFs, make web requests and so on. Scripts in GLAMpipe's nodes can pass parameters to cores which then return the result of operation back to node.