A Distributed Computing Network

Sandstorm Demo

It’s hard to show you Sandstorm, so allow us to tell you about it.

Users Submit a Campaign to Sandstorm

Campaign Data: 20170918 podcast/38 episode.137

Campaign Function:

function run(url, latestEpisode) {
  // fetch feed url
  // parse and normalize the feed
  // find episodes newer than latestEpisode
  // build JSON object with episodes
  // return episodes JSON

Sandstorm Adds the Campaign to Job Queue

Language Data Function
Javascript 20170918 function run(url, latestEpisode) { ... }
Javascript podcast/38 function run(url, latestEpisode) { ... }
Javascript episode.137 function run(url, latestEpisode) { ... }

The Sandstorm Network  Starts Running Jobs

The job runners in The Sandstorm Network that best fit the needs of the job are notified of new jobs, and then they start running the jobs. Each job is run, its result sent back, and another job is requested… until the campaign is completed.

Jobs written in Node, Ruby, Python, or Go are sent to our network of servers with language specific containers. Jobs written in Javascript are sent to our network of web browsers and native apps (iOS, Android, macOS).

Results Are Sent to The User

Results can be sent to the user directly or written to one of their S3 buckets.

Kinds of Campaigns: Rivers, Lakes, Spirals

RIVERS are campaigns where there is an ongoing stream of data as input. Sandstorm keeps processing the data for as long as it comes in. Typically, the results for these kinds of campaigns are streamed back, but aggregate reports can be generated too on some schedule (hourly, daily, etc).

LAKES are campaigns where there is a finite set of big data as input. Sandstorm processes all the data and sends and returns a single report or a finite stream of results. Once it’s done, it’s done.

SPIRALS are campaigns that are similar to Lakes in that they start with a finite set of big data. The difference is that the results from the first run of the campaign become the input for the second run. The output of the second is the input of the third, and so on. These are good for jobs that need knowledge of the previous run to get accurate results, like fetching new episodes in a podcast feed.

Start Using Sandstorm

We’re currently looking for beta customers. Have a problem that needs solving with on-demand distributed computing? Let us build something for you! Get in touch, we’d love to talk with you.