Automate steps to simplify tasks

Pipelines are awesome for task automation

What are tasks, pipelines and steps?

We define the term task quite broadly, within the digital realm:

Some work that involves a computer

Pipelines are a relatively new technology that is backed by open standards. They offer an elegant way to break larger tasks into steps that can work together.

The Automate app in Tag has an easy to use no-code pipeline editor. The range of tasks it can handle using built-in steps is surprising. All of this power (which until now only programmers had) is now at the fingertips of any desktop computer user.

The real fun happens when you connect steps together. For example, you could use three steps to: read a file, process it somehow (see below), and save the results to a new file. Next you could swap out the first step, and instead call a web API or read from a database. It's a bit like Lego for instructions to control the computer.

What kind of tasks?

The majority of available steps are defined by the public XProc standard. These steps cover a wide range:

  • Read and save files

  • Read from or post to web APIs

  • Convert between text, XML and JSON content types

  • Zip/unzip and compress/uncompress files

  • Search in content to modify with precision

  • Edit, delete, insert and replace

  • Compare, count and filter content

  • Join or merge JSON files

  • Generate labels for repeating items

  • Process text files in clever ways

  • Run powerful XML queries

  • Generate data or content using a rich set of XSLT instructions

  • And more...

Tag also supports file and validation steps defined by open standards:

  • List files and folders within a folder

  • Copy, delete or move files

  • Create temporary files

  • Create a folder

  • Validate files

  • And more...

An important feature of XProc is that custom steps can be used as well. Tag implements several custom steps that extend the range of instructions available to pipelines:

  • Call a secure web API connector

  • Open a CSV file and convert it to XML

  • Save (generated) word processing content as *.docx files

  • Convert JSON to/from XML in a simplified way

  • Call a SQL database

  • Call a SPARQL ontology / knowledge graph

  • Prompt the user for input to a running pipeline

  • Call Google API using OAuth 2.0 authorization

  • Call powerful machine learning algorithms (contact us for more info)

  • And more...

The above steps can be mix and matched in many ways. The number of possible combinations is hard to count.

The Automate app in Tag simplifies how you connect steps using a no-code editor. It hides complexity and only displays what you need, rather than overwhelm with all options all the time.

Note that the Automate app can also view and edit many other files types. In particular, you can use it to generate content that may or may not be rich text (the Scribe app only generates rich text). Topics in the Tag feature map discuss this further, including the generation of (SVG) vector graphics.

Tag's Automate app works with many different file types. The most powerful of these is pipeline files, because pipelines can in turn control other files.

The pipeline editor uses a graph at the top to show how steps connect together. You can quickly jump to any step visible in the tree. For any two adjacent steps, by default, the output of the step above flows into the step below. For example in the displayed pipeline, a web API is called and the result is stored in a file.

Pipeline Editor

It takes all kinds

Since pipelines use other files and services, you can imagine them at the center of lots of other technology widely in use today.

Content and data can be pulled from many different sources. It can be processed in its original format (usually text, XML or JSON) or converted to another.

Processing can involve logic defined by the pipeline, or instructions defined by another file type (e.g., XSLT transformations or SVG graphics). It can also be delegated to a web API or machine learning algorithm (e.g., optical character recognition, translation, entity detection).

When the task is done the results can be stored in a file, sent to a web API, stored in a database or fed into another pipeline.

Working with other pipeline engines

Open standards like XProc are great because you can rely on more than one software provider. Any pipeline created in Tag (using standard steps), can be run in any XProc engine (e.g., Morgana or Calabash).

Tag's Automate app goes even further and allows you to run pipelines in the editor using an external (Java compatible) XProc engine. This is necessary to run steps that Tag does not yet natively implement. For this reason, Tag embeds a recent version of Morgana ensuring that you can access 100% of XProc functionality.

Note that a common use case for pipelines is within a web server application. Tag's XProc engine is for desktop use only within Tag, and does not support server applications. However, you can edit pipelines in Tag and deploy them on a server that has Morgana or Calabash installed.

The Automate app can be used for free if you are happy editing raw XML instructions. If instead you prefer to use the no-code editor (who wouldn't), you need to have a Starter or other paid subscription. Purchasing a subscription will encourage us to improve our no-code tools, and continue making pipelines more intuitive and productive to use.