Handling Race Conditions in Rails with Fencing Tokens – One of many ways

Concurrency is a common feature sought out by all developers and engineers in an application, and this is an age-old concept that we have been bettering constantly through the years. In it’s simplest form, concurrency is multitasking, and when more than one task runs at the same time in a system, there are bound to be performance issues of one or many sorts. Rails handles the concurrency actions effectively using Fencing Tokens.

Basically, we want an application or system or machine to do multiple things at the same time. We want our code to do more things in lesser time and every developer will want to increase code throughput at the core. In such concurrent or multi-threaded environments, where multiple users are using the deployed application or webserver, we will at some point, inevitably encounter race conditions.

So what is a race condition?

A race condition is a situation in which the output of the code becomes dependent on the sequence or timing of other uncontrollable events; which means that the if the events of the program do not take place in the order as intended by the developer, it can cause an error or become a bug! The order in a race condition becomes utmost important.

Let us take an example in which multiple users need to update the database. Rails does provide ActiveRecord locking in the database level, as well as developers can perform validation in the application level. But even with these approaches there can be situations where the data integrity is compromised (and this is out of scope for this post!) and . In this post, we will see how we can use the concept of fencing tokens to effectively handle such situations.

Let us consider a simple situation where a user updates records in a store that will be used my multiple workers. Each book has a title and price, and is queued to be processed.

Fencing Tokens Rails - book

– id: 1

title: The_Secret

price: 10

 

– id: 2

title: The_Winner

price: 15

 

– id: 3

title: The_External

price: 20

 

And if there is sudden burst of updates to the same book, the records will be something like:

 

– id: 1

title: The_Secret

price: 10

 

– id: 2

title: The_Winner

price: 15

 

– id: 1

title: The_Secret

price: 12

 

– id: 1

title: The_Secret

price: 15

 

When multiple workers try to access this concurrently, and if the third worker for instance is slightly slower in processing, we would end up with the wrong data in the system, the price of The_Secret being 12 updated finally.

To avoid such a situation, we shall use the fencing tokens in this case to see how we can solve this. We simply create a token with the book and increment it for every update of the records.

 

– id: 1

title: The_Secret

price: 10

fencing_token: 1

 

– id: 2

title: The_Winner

price: 15

fencing_token: 2

 

– id: 1

title: The_Secret

price: 12

fencing_token: 3

 

– id: 1

title: The_Secret

price: 15

fencing_token: 4

 

Now, each time we push the book to the queue, the fencing token is sent along with the other attributes. Here, once processed by a worker, we key the fencing tokens with the id attribute of the book and store it.

Everytime the book is picked up for processing, it checks if the fencing token is greater than the fencing token stored for that book in the table. If it is not, it simply ignores the update. This works perfectly in our example, since when worker 3 processes the book, it sees that the fencing token 3 is lesser than the one already updated (whose fencing token is now 4), and simply does not perform the update. Hence the price is left as 15 for The_Secret and is not overwritten by 12.

We use fencing tokens by generating these fencing tokens, create initializers and define a class that increments the token.

Now, let us see how we can generate fencing tokens in Rails:

The token is an integer and we can add a name to it so that we have different tokens for different queues.

class CreateFencingTokens < ActiveRecord::Migration #recordtoken

def change

create_table :fencing_tokens do |t|

string :name

integer :token, default: 1

end

 execute(“insert into fencing_tokens (name, token) values (‘books’, 1)”)

end

end

Since we may have multiple tokens with this method, we will have to use an initializer as:

# config/initializers/fencing_tokens.rb

if ActiveRecord::Base.connection.table_exists?(:fencing_tokens) #record

%w( postings foobars ).each do |name|

unless FencingToken.find_by(name: name)

FencingToken.create!(name: name)

end

end

end

Next, we define a class in order to increment the token and return it to us, and we want it done atomically. Therefore, we do it using SQL.

class FencingToken < ActiveRecord::Base

validates :name, uniqueness: true

def self.generate(name)

raise ArgumentError, “invalid name #{name}” unless /\A[a-z,_]+\z/ === name #validation

sql = “update fencing_tokens set token = token + 1 where name =  ‘#{name}’ “\’returning token’

result = ActiveRecord::Base.connection.execute(sql)

result[0][‘token’].to_i

end

end

Here, we are just reassuring that there is no injection risk since we are directly using a string. We could have done a simpler class such as:

class FencingToken < ActiveRecord::Base

def self.generate(name)

ft = find_by(name: name)        # BAD of BAD to BAD

ft.update(token: ft.token + 1)  # NOOOOOO Dont do it !!!1

ft.token

end

end

but this would have resulted in ambiguity again, since if there are multiple threads, another active thread could access the token between the read and write calls of this class execution, and therefore defy the purpose of the fencing tokens. This is why it is better to do some SQL directly. So now we can just call the FencingToken.generate(‘books’) class to atomically increment the token.

There are other methods to handle race conditions. We could use the timestamps generated with the record updates by the database itself (which will be a similar concept of atomic updates and act as a unique key), but then again, it would certainly be near to impossible to be accurate in a distributed system environment. In such an environment, there is no way to be sure that all systems involved will have accurate and synchronized time set.

Again, as with all concurrency solving methods, this one does not address all possible issues. There still could be a situation in which this method could not work perfectly. There could be some performance issues in a large system where the fencing token table could be serious contention, or when multiple threads access the same token. But for most similar situations, this is pretty much a to-go solution.