Amazon S3 is a extremely powerful service at the core of Amazon Web Services. However, outside of a production environment, S3 can be challenging to work with. It involves passing keys around, provisioning user accounts, and a reliable network connection — not to mention it costs money.

At Spool, we built Fake S3 to make working with S3 in development and testing environments much easier. Our goal was to make a self contained executable that can mimic the majority of S3 Rest API with few external dependencies.

For development, each engineer runs her own instance of Fake S3 where she can put gigabytes of images and video to develop and test against, and her setup will work offline because it is all local. We also have a continuous integration setup that is running tests 24/7 (often against large video files). Fake S3 saves us $1000 a month in bandwidth alone for our tests. In both development and testing, the time saved in not waiting for assets to go back and forth into AWS, especially our larger media files, makes Fake S3 very useful.

We’re releasing Fake S3 as a gem on github. It’s an early release and we’ll keep improving it. If you have ideas or issues, please contribute to the project!

Installation

gem install fakes3

Running

fakes3 -r ~/fakes3_root -p 10001

Example Client Code

require 'rubygems'
require 'aws/s3'

include AWS::S3
AWS::S3::Base.establish_connection!(:access_key_id => "123",
                                    :secret_access_key => "abc",
                                    :server => "localhost",
                                    :port => "10001")

Bucket.create('mystuff')

('a'..'z').each do |filename|
  S3Object.store(filename, 'Hello World', 'mystuff')
end

bucket = Bucket.find('mystuff')
bucket.objects.each do |s3_obj|
  puts "#{s3_obj.key}:#{s3_obj.value}"
end

Bucket.delete("mystuff",:force => true) # Delete your bucket and all its keys

In general clients can work if you can specify a host and port to connect them, as well as forcing path style requests (instead of subdomains.) Subdomain style S3 requests can work, but it involves adding your bucket names into /etc/hosts (ie. s3.localhost or mybucket.localhost) or using dnsmasq if you have a large number of buckets.

Simulating Network Conditions

Another useful feature is the support for simulating network conditions. You can run fakes3 with bandwidth limiting enabled like so:

fakes3 -r ~/fakes3_root -p 10001 --limit=50K

This will limit your GET request bandwidth to 50K/s per request instead of instantly off of your local machine. This is very convenient, for example, to simulate how mobile devices would behave in the real world.

Related Tools

Fake S3 is great for development and testing due to its simplicity, but is not intended to replace S3 in production. If you want to replace S3, there are other tools such as Ceph, ParkPlace (supports bitorrent), Boardwalk (S3 interface in front of MongoDB), and RiakCS that you can check out.

We’re very proud to announce that we’ve raised $1 million in outside financing for Spool. Our complete list of investors is below.

Though we’ve only been in private beta for 3 months, we’ve seen tremendous growth and received an overwhelmingly positive response. We have raised money to continue building a product our users love and are humbly looking forward to the opportunity to do so. We’ll be investing heavily in improving SpoolBot, polishing our mobile and browser applications, making the product easier to use for new users, and making it easier to share content with your close friends.

In the mold of great businesses we admire, our goal is to efficiently use our outside investment to improve the lives of millions of users. To this end, we have selected investors who have experience investing in the highest growth technology companies (Google, Twitter, Facebook, and more), who have themselves founded or built world changing companies (YouTube, Palantir,  LinkedIn, and more), and entrepreneurs who have relevant experience outside of technology companies as well.

We have some big things in store for Spool and are looking forward to sharing them with you. As always, we’d love to hear your thoughts and feedback — please don’t hesitate to email us at feedback@getspool.com.

-The Spool Team

Our Investors

• SVAngel
• Felicis Ventures
• Start Fund
• Charles River Ventures
• Vivi Nevo
• Steve Chen
• Elad Gil
• Deep Nishar
• Kevin Donahue
• Joe Lonsdale
• Bill Lohse
• David King
• Nils Johnson
• Matt Ocko
• Raymond Tonsing

In case you’re wondering how to save to Spool via email, read this post

As always, if you have any questions, please email us at support@spoolbot.com

SpoolBot is experiencing tremendous load right now. We apologize but it may take a bit longer than normal to get your recordings as SpoolBot makes its way through the backlog. We’ve increased the number of SpoolBots by 5x to handle the increase in traffic and are adding more by the minute.

Thanks for your patience! We’ll working very hard right now to find ways to get you your recordings as quickly as possible.

- The Spool Team

You can now send an email with a URL or an attachment to your personal SpoolBot email address. SpoolBot will put the contents of up to 5 URLs or any number of attachments into your Spool account!

Add your Spool to the contacts on your phone/tablet

Find your personalized email address in settings

• numerous improvements to our algorithms that detect multipage articles
• better handling of javascript click handlers on content pages
• SpoolBot preserves on-page anchor tags (useful for table of contents, footnotes, etc.)
• support for file links (a link to a .jpg file or a .mp3 file)
• audio file support — SpoolBot can now send down audio files to clients. SpoolBot don’t extract audio from webpages yet, but SpoolBot knows how to deal with the data.
• lots of performance updates.

We’re pushing SpoolBot updates every few days so keep hammering on Spool and we’ll keep teaching SpoolBot to be smarter.

Crash Course on Bitmap and Redis Bitmaps

Bitmap (aka Bitset)

A Bitmap or bitset is an array of zeros and ones. A bit in a bitset can be set to either 0 or 1, and each position in the array is referred to as an offset. Operations such as logical AND, OR, XOR, etc. and other bitwise operations are fair game for Bitmaps.

Population Count

The population count of a Bitmap is the number of bits set to 1. There are efficient algorithms for calculating population count. For instance, the population count of a 90% filled bitset containing 1 billion bits took 21.1 ms on a MacBook Pro. There is even a hardware instruction in SSE4 for the population count of an integer.

Bitmaps in Redis

Redis allows binary keys and binary values. Bitmaps are nothing but binary values. The setbit(key, offset, value) operation, which takes O(1) time, sets the value of a bit to 0 or 1 at the specified offset for a given key.

A simple example: Daily Active Users

To count unique users that logged in today, we set up a bitmap where each user is identified by an offset value. When a user visits a page or performs an action, which warrants it to be counted, set the bit to 1 at the offset representing user id. The key for the bitmap is a function of the name of the action user performed and the timestamp.

In this simple example, every time a user logs in we perform a redis.setbit(daily_active_users, user_id, 1). This flips the appropriate offset in the daily_active_users bitmap to 1. This is an O(1) operation. Doing a population count on this results in 9 unique users that logged in today. The key is daily_active_users and the value is 1011110100100101.

Of course, since the daily active users will change every day we need a way to create a new bitmap every day. We do this by simply appending the date to the bitmap key. For example, if we want to calculate the daily unique users who have played at least 1 song in a music app for a given day, we can set the key name to be play:yyyy-mm-dd. If we want to calculate the number of unique users playing a song each hour, we can name the key name will be play:yyyy-mm-dd-hh. For the rest of the discussion, we will stick with daily unique users that played a song. To collect daily metrics, we will simple set the user’s bit to 1 in the play:yyyy-mm-dd key whenever a user plays a song. This is an O(1) operation.

redis.setbit(play:yyyy-mm-dd, user_id, 1)

The unique users that played a song today is the population count of the bitmap stored as the value for the play:yyyy-mm-dd key.To calculate weekly or monthly metrics, we can simply compute the union of all the daily Bitmaps over the week or the month, and then calculate the population count of the resulting bitmap.

You can also extract more complex metrics very easily. For example, the premium account holders who played a song in November would be:
(play:2011-11-01 ∪ play:2011-11-02 ∪...∪play:2011-11-30) ∩ premium:2011-11

Performance comparison using 128 million users

The table below shows a comparison of daily unique action calculations calculated over 1 day, 7 days and 30 days for 128 million users. The 7 and 30 metrics are calculated by combining daily bitmaps.

Optimizations

In the above example, we can optimize the weekly and monthly computations by caching the calculated daily, weekly, monthly counts in Redis.

This is a very flexible approach. An added bonus of caching is that it allows fast cohort analysis, such as weekly unique users who are also mobile users — the intersection of a mobile users bitmap with a weekly active users bitmap. Or, if we want to compute rolling unique users over the last n days, having cached daily unique counts makes this easy — simply grab the previous n-1 days from your cache and union it with the real time daily count, which only takes 50ms.

Sample Code

A Java code snippet below computes unique users for a given user action and date.

import redis.clients.jedis.Jedis;
import java.util.BitSet;
...
  Jedis redis = new Jedis("localhost");
...
  public int uniqueCount(String action, String date) {
    String key = action + ":" + date;
    BitSet users = BitSet.valueOf(redis.get(key.getBytes()));
    return users.cardinality();
  }

The code snippet below computes the unique users for a given given user action and a list of dates.

import redis.clients.jedis.Jedis;
import java.util.BitSet;
...
  Jedis redis = new Jedis("localhost");
...
  public int uniqueCount(String action, String... dates) {
    BitSet all = new BitSet();
    for (String date : dates) {
      String key = action + ":" + date;
      BitSet users = BitSet.valueOf(redis.get(key.getBytes()));
      all.or(users);
    }
    return all.cardinality();
  }