Posts Tagged ‘Memcached’

Storing hundreds of millions of simple key-value pairs in Redis

November 2, 2011 Leave a comment

We needed a solution that would:

  • Look up keys and return values very quickly
  • Fit the data in memory, and ideally within one of the EC2 high-memory types (the 17GB or 34GB, rather than the 68GB instance type)
  • Fit well into our existing infrastructure
  • Be persistent, so that we wouldn’t have to re-populate it if a server died

One simple solution to this problem would be to simply store them as a bunch of rows in a database, with “Media ID” and “User ID” columns. However, a SQL database seemed like overkill given that these IDs were never updated (only inserted), didn’t need to be transactional, and didn’t have any relations with other tables.

Instead, we turned to Redis, an advanced key-value store that we use extensively here at Instagram (for example, it powers our main feed). Redis is a key-value swiss-army knife; rather than just normal “Set key, get key” mechanics like Memcached, it provides powerful aggregate types like sorted sets and lists. It has a configurable persistence model, where it background saves at a specified interval, and can be run in a master-slave setup.

via Instagram Engineering • Storing hundreds of millions of simple key-value pairs in Redis.


Differences between memcached and redis?

October 27, 2011 Leave a comment

Source: Adam D’Angelo on What are the differences between memcached and redis? – Quora.