Programming Languages & Frameworks
Ruby on Rails
The primary driver of our web applications is Ruby on Rails, which we have found to give us the best tooling to develop well-tested, scalable and reliable web applications. Its MVC-based nature (which inspired numerous clones in other languages) and mature library allow us to quickly iterate on ideas and bring them to our customers.
In many of our background tasks (primarily scraping and some data processing) we also started to more extensively utilize Node.js. Similarly to ruby it gives us a great library of code to include and utilize.
Python is primarily used in data science, but also has some presence in legacy background scraping tasks we still maintain and run.
Our storage solutions are as versatile as the data for which we use them. Primarily we are focused on three storage databases for long term data storage (discounting redis, memcached etc).
For general purpose relational data that is used to keep our sites working, we rely on MySQL. As a well established and documented database backend with good tooling, we have found it to be adequate for all our work so far.
To efficiently make our data searchable, we utilize Elasticsearch which we run on multiple servers as a distributed data set. It allows us to search through our big data sets with ease.
For storage of big multi-million document data collections, we utilize MongoDB. It serves as the backend for the whole of our extensive data collection, for which it is perfectly suited due to its web-scale architecture and design.