Open-Source/Free

.Net Developer's Guide to Getting Started with Cassandra

Eden Ridgway — Thu, 05 Nov 2009 22:32:51 GMT

I've been quite curious about the various BigTable like implementations as they could have some bearing on what I am involved with at work. After reading various write ups about the competing approaches (e.g. HyperTable, HBase, Dynomite, Voldemort, Mongodb) I decided that Cassandra looked like the best option for what I'm after. What I didn't realise was that it wasn't going to be as straight forward as it appears to set it all up as there is no end-to-end guide of getting started with it. So the objective of this post is not to explain Cassandra but save you several hours of jumping all over the web trying to work out what you should do next and how to generate the necessary C# Cassandra Thrift client code. If parts of the post don't apply, e.g. VM setup, simply skip ahead to the parts that do. If you want more information about Cassandra you should check out the following posts:

So to get Cassandra up and running in a Virtual Machine (or physical if you choose) you should follow the following steps (valid as at 5 Nov 2010):

Create a virtual machine to host Ubuntu 9.10 x86 Desktop. I use VirtualBox as my VM host on windows so I will refer to this here but feel free to substitute it with you favourite VM software. The VM I created has 384MB allocated to it with an 8GB virtual hard drive using a bridged network adapter (so it can be accessed from the Windows XP host). Once you've gone through the install process install the client tools so that you change the screen resolution, share folders, etc. To do this with virtual box go to Devices > Install Guest Additions and then open up a terminal window and execute

sudo /media/cdrom/VBoxLinuxAdditions-x86.run

Once this is complete you will need to reboot your VM.
Now it is time to install Cassandra/Thrift prerequisites and software that we will to set everything up:

sudo apt-get install libboost-dev automake libtool flex bison pkg-config g++ -y
sudo apt-get install sun-java6-jdk -y
sudo apt-get apt-get install ant -y
Once that is done we can install other useful Ubuntu software that we will use:

sudo apt-get install rapidsvn -y
sudo apt-get install monodevelop -y

I also like to install the following:

sudo apt-get install nautilus-open-terminal -y
sudo apt-get install samba smbfs smbclient winbind -y
sudo apt-get install nautilus-share -y
You may discover that there is an issue with your Java runtime home directory. If that occurs, select Sun's Java Runtime as the default Java version by running the following command from the terminal window and selecting the third option:

sudo update-alternatives --config java
Now open up Nautilus by going to Places > Home Folder. Right click and create a Development folder. Within the Development folder create another two folders: Cassandra and Thrift.
We can now get the source for Cassandra and Thrift. Run RapidSVN by going to Applications > Programming > RapidSVN. Check out the source from SVN by going to the RapidSVN menu option Repository > Checkout. In the URL enter http://svn.apache.org/repos/asf/incubator/thrift/trunk and for the Destination Directory browse to the Thrift development folder. So it should look like this:

Now get the Cassandra code by doing the same thing but entering http://svn.apache.org/repos/asf/incubator/cassandra/trunk as the URL and selecting the Cassandra directory.
Once we have all the code we are ready to compile Cassandra, so let's do that by executing the following commands from the terminal window:

cd ~/Development/Cassandra
ant
Start the Cassandra server by opening a new terminal window and running the following commands:

sudo ~/Development/Cassandra/bin/cassandra -f

The running server will look like the screenshot below. You should take note of the thrift binding address of localhost/127.0.0.1:9160.
Now we want to get communicating with the server, so we will need to compile Thrift so that we can use it to generate our C# client:

cd ~/Development/Thrift
./bootstrap.sh
./configure
make
sudo make install
Now we can finally generate a C# client for the Thrift calls to Cassandra:

cd ~/Development/Cassandra/interface/
~/Development/Thrift/compiler/cpp/thrift -gen csharp cassandra.thrift

This will create the following directories inside of the Cassandra/interface folder: gen-csharp/Apache/Cassandra. These files along with the Thrift.dll will be copied into a new demo project we will create using MonoDevelop in the Development folder. So again in the terminal window execute:

mkdir ~/Development/CassandraDemo
cp ~/Development/Cassandra/interface/gen-csharp ~/Development/CassandraDemo -R
cp ~/Development/Thrift/lib/csharp/Thrift.dll ~/Development/CassandraDemo
Open up MonoDevelop and create a new console application by going to File > New > Solution. Now do the following:
1. Select Console project
2. Enter CassandraDemo as the name
3. For the location browse to the Development folder that we created in your home directory.
4. Uncheck the 'Create separate Solution directory' check box.
5. Click the Forward button
6. Don't make any changes on the next screen and just click Ok.
7. Right click on the CassandraDemo project in the left tree view and select Add > Add Files.... Navigate to ~/Development/CassandraDemo/gen-csharp/Apache/Cassandra and select all the files in the folder. You will now have a project that looks like this:
8. Add a reference to the Thrift assembly by right clicking on References and selecting Edit References. Change to the .Net Assembly tab and double click on the Thrift.dll and click OK.
9. Now we can add the sample code to test it all out. So double click on Main.cs to ensure that is the class in focus in the editor. Replace the contents of the file with this:
  
  namespace CassandraDemo
  {
      using System;
      using System.Collections.Generic;
      using System.Diagnostics;
  
      using Apache.Cassandra;
      using Thrift.Protocol;
      using Thrift.Transport;
  
      class Program
      {
          static void Main(string[] args)
          {
              TTransport transport = new TSocket("localhost", 9160);
              TProtocol protocol = new TBinaryProtocol(transport);
              Cassandra.Client client = new Cassandra.Client(protocol);
  
              Console.WriteLine("Opening connection");
              transport.Open();
  
              System.Text.Encoding utf8Encoding = System.Text.Encoding.UTF8;
  
              long timeStamp = DateTime.Now.Millisecond;
              ColumnPath nameColumnPath = new ColumnPath()
                                          {
                                              Column_family = "Standard1",
                                              Column = utf8Encoding.GetBytes("name")
                                          };
  
              Console.WriteLine("Inserting name columns");
  
              //Insert the data into the column 'name'
              client.insert("Keyspace1",
                            "1",
                            nameColumnPath,
                            utf8Encoding.GetBytes("Joe Bloggs"),
                            timeStamp,
                            ConsistencyLevel.ONE);
  
              client.insert("Keyspace1",
                            "2",
                            nameColumnPath,
                            utf8Encoding.GetBytes("Joe Soap"),
                            timeStamp,
                            ConsistencyLevel.ONE);
  
              //Simple single value get using the key to get column 'name'
              ColumnOrSuperColumn returnedColumn = client.get("Keyspace1", "1", nameColumnPath, ConsistencyLevel.ONE);
              Console.WriteLine("Column Data in Keyspace1/Standard1: name: {0}, value: {1}",
                                utf8Encoding.GetString(returnedColumn.Column.Name),
                                utf8Encoding.GetString(returnedColumn.Column.Value));
  
              Console.WriteLine("Getting splice range");
  
              //Read an entire row
              SlicePredicate predicate = new SlicePredicate()
                                         {
                                            Slice_range = new SliceRange()
                                                          {
                                                              //Start and Finish cannot be null
                                                              Start = new byte[0],
                                                              Finish = new byte[0],
                                                              Count = 10,
                                                              Reversed = false
                                                          }
                                         };
  
              ColumnParent parent = new ColumnParent() { Column_family = "Standard1" };
              Dictionary<string , List<ColumnOrSuperColumn>> results = client.multiget_slice("Keyspace1",
                                                                        new List<string>() { "1", "2"},
                                                                        parent,
                                                                        predicate,
                                                                        ConsistencyLevel.ONE);
  
              foreach (KeyValuePair<string, List<ColumnOrSuperColumn>> resultPair in results)
              {
                  Console.WriteLine("Key: {0}", resultPair.Key);
  
                  foreach (ColumnOrSuperColumn resultColumn in resultPair.Value)
                  {
                      Column column = resultColumn.Column;
                      Console.WriteLine("name: {0}, value: {1}", utf8Encoding.GetString(column.Name), utf8Encoding.GetString(column.Value));
                  }
              }
  
              Console.WriteLine("Closing connection");
              transport.Close();
          }
      }
  }
10. Hit F5 and watch the Application Output window. This is what you should see:

Are Database Migration Frameworks Worth the Effort?

Eden Ridgway — Sat, 03 Jan 2009 11:52:03 GMT

I’ve always used SQL scripts to manage database creation and migration. At my current company we have a database project with patch scripts that conditionally execute based on the database version. These scripts are simply run in sequence by an installer which stores the current database version as a database extended property. The approach relies on version based naming of SQL scripts and developers to remember to update the patch script whenever they make a structure or stored procedure change. Of course one could try to use a tool such as Redgate SQL compare to generate diff patch scripts but that won’t always suffice as that won’t capture refactoring such as column name changes or data migration (for more read The Problem with Database Diffs).

The use of SQL scripts is not without it’s problems. The most annoying of these is that a patch script may break as SQL will parse and validate your script against the existing table structure even if the statements are not going to run due to version check condition. This forces one to use nasty EXEC/sp_execute alter/create statements with double escaped strings (Yuck!). I find the following particularly problematic with this approach:

The string based SQL statements will always pass validation even though when they are executed they may be completely screwed
It is impractical to generate these by hand and the best way to create them is to use SQL Management Studio.
If your stored procedures are not wrapped in strings, to update the patch script requires you to script out the scripts from the database. Basically this increases development friction and discourages updating of the patch script.
It becomes difficult to determine when a structural change was made as one has to trawl through huge SQL scripts and it is not easy to use a regex search to find a change.

Hence I long for something a little more strongly typed, which is why the growing number of .Net based Database Migration frameworks have caught my eye. I want to be sure that there isn’t something obvious that I was missing out by not using one. Fortunately Ben Scheirman has done a fair amount of research already and wrote a fairly good round up of available .Net migration tools. Looking at the tools there are clearly two different types of frameworks available:

Structured SQL script execution – These tools provide a framework under which SQL scripts will be executed to create and upgrade a database to a specific version. Examples of these are: Tarantino and LiquiBase.
Rails like Active Record Migration – Commands to upgrade or downgrade the database schema and data can be coded in a .Net language, with the possibility of executing SQL statements if required. Tools that fall into this category are: RikMigrations, SubSonic Migrations, Machine.Migrations and Migrator.Net.

The obvious question that springs to mind when looking at these frameworks is whether or not the additional development friction (additional tasks that must be performed during the course of development) is worth the benefit. To be honest I don’t see a massive difference between the structured SQL script execution approach and what I’ve been doing all along. However the likes of LiquiBase do provide a functionality that nicely captures database refactorings and diffs, so maybe it could be used as more of a general toolset. However the Active Migration approach would be a more radical departure so I’d like to explore that more.

Here are my thoughts on Active Record Migration like frameworks:

Best paired with an ORM framework, such as NHibernate or Subsonic, where one is not writing a lot of custom SQL and hence the the upgrade classes can be kept clean and focus on structural and data migration. That is not to say that there will not be benefit to having them execute a lot of SQL statements to add/update stored procedures, it is just that it may get a little messy and possibly tedious.
Using C# code to do database change would provide an opportunity to use something like log4net to dump out progress information. I've always longed for a decent way of doing this that doesn't involve a custom proc that inserts into a table. I really like the idea of having a degree of compile time checking for the database changes. Of course with SQL scripts you could always run them on each build and if they break, you fail the build.
The whole idea of downgrading a database scares me because if someone accidently points an old version of an application (with automatic migration logic) to the database (think xcopy deployment) you could end up losing tables and data. Of course one should always backup a database before doing a deployment, but I’m talking about a situation where one doesn’t know it is pointing to the wrong database until it is too late. I hope these tools backup the database before applying changes. In my mind a downgrade should only be performed if a rollout fails and one has to revert to a backup taken before the upgrade was undertaken.
I really like the idea of using anonymous objects to insert data into the database (RikiMigration supports this) and avoiding the need to write insert statements.
Do you not run into potential development concurrency and structuring issues? So for example during a development cycle does one dump all changes into one class or into several?
The database vendor abstraction that is can provide is only attractive to a small number of projects. Furthermore one invariably will need to execute vendor specific statements and is best suited to a multi-vendor ORM such as NHibernate.
The approach may not fly in an organization where a DBA must review all SQL changes. Although I don’t see any reason why the frameworks couldn’t generate these for you.

I think if I was currently working on a project that used an ORM like NHibernate or LINQ to Entities, the migration frameworks would be a lot more appealing. However with other types of projects I’m worried that the additional effort isn’t really going to be worth the benefit.

Am I missing any other potential benefits (or problems)? I would really like to hear if there are other compelling reasons to adopt the approach.

An Updated LINQ to WMI Implementation

Eden Ridgway — Wed, 02 Jan 2008 07:51:56 GMT

I recently needed to do some WMI queries so I went hunting for a LINQ implementation. I found a great basic implementation that Emile Bosch did way back in 2005. Once you have finished reading this post you should view his blog post for more detail on the implementation.

Unfortunately Emile's WMILINQ implementation no longer works as it was based on a beta version of the framework so I brushed it off and upgraded it to work with the RTM release of .Net 3.5. I also made several other other improvements such as:

Updated the code generator to identify and generate array properties and emit a generated code attribute on the associated class
Fixed a bug with string where filter parameters
Improved the CIM Type mapping conversion logic
Added array mapping support to the LINQ query
Changed the name to match the new Microsoft LINQ to X naming convention

Once I had it working I could execute queries against the WMI objects like this:

var query = from account in context.Source<Win32_UserAccount>()
select account;

foreach (Win32_UserAccount account in query)
{
Console.WriteLine(account.Name);
}

Doing basic queries with the provider is really easy, but be aware that the implementation is very limited. It does not support projections, nested queries and referenced variables. So for example the following query generates an incorrect WQL statement.

string name = "Me";

var query = from account in context.Source<Win32_UserAccount>()
where account.Name == name
select account;

One would expect it to generate this statement:

SELECT * FROM Win32_UserAccount WHERE name = 'Me'

but instead executes this:

SELECT * FROM Win32_UserAccount WHERE name = 'name'

Regardless of these shortcomings I still found it quite useful, and I wanted to share the upgraded version with you. So if you'd like to grab it, here's the download.

I am currently working on a rewrite of the implementation using Matt Warren's excellent set of Building an IQueryable Provider blog posts as a basis. So far it is not looking too bad and it overcomes several of the problems with this implementation, but it requires quite a bit more effort to finish it off, so it may be a while before I release it.

Profiler for SQL Express

Eden Ridgway — Tue, 10 Jul 2007 04:45:00 GMT

For those of you who are doing things on the cheap and only using SQL Express, you'll be glad to know that there is an open source SQL Express profiler project. As one would expect it is pretty basic, but has all the important features, like the ability to filter and select the events and columns to trace. I had a look around the code and it is pretty simple (it could do with some more comments though ), so it should be easy enough to extend should you want to.

Processing CSV Data

Eden Ridgway — Mon, 19 Feb 2007 05:52:13 GMT

If you are a .Net developer, there is very little reason why you should ever write your own CSV parser. SecretGeek has a pretty good blog post on why it is a bad idea. He mentions a library I have known about for a while called FileHelpers. It has some pretty awesome features, such as:

Serialization and deserialization of delimited or fixed width data
Attribute based field conversion definition (e.g. [FieldConverter(ConverterKind.Date, "dd-MM-yyyy")] ). You can also create custom converters.
File differences for files with the same record structure
The ability to skip empty or first/last N records

An alternative less powerful approach is to use Microsoft's XmlCsvReader. This is more useful if you want to treat the file like a read only XML document.

Safari on Windows (Web Kit)

Eden Ridgway — Tue, 23 Jan 2007 05:26:11 GMT

There has been quite a bit of talk about getting a Windows version of Safari. Well with the Web Kit and Swift projects this seems to be finally happening. It is still early days and the Swift browser crashed on me several times but it has promise. Since I don't have a Mac handy it at least gives me a glimpse of how a web site might look/operate in Safari.

What made me even more interested in the Swift browser was that it is written in C#, but unfortunately the subversion repository url provided does not work. The good news is that the Web Kit project seems better organised and I didn't have a problem getting hold of that code. It also has pretty good instructions on how to build it on Windows.

Since the Swift wiki does not have a screenshot of the browser, I thought I'd provide one:

iTextSharp XML to PDF Example

Eden Ridgway — Sun, 31 Jul 2005 14:36:54 GMT

The iTextSharp's current XML to PDF example is clearly out of date and does not work. So I went about fixing the example. You can download it here: iTextSharpExample.zip.

The generation of a PDF file from XML is surprisingly painless. The steps are as follows:

Create an iTextSharp document
Document document = new Document();
Get an instance of the PDF writer and at the same time specify the output PDF
PdfWriter.GetInstance(document, new FileStream("ExampleDoc.pdf", FileMode.Create));
Create an iTextSharp Xml TextHandler passing in the document instance
ITextHandler xmlHandler = new ITextHandler(document);
Get the TextHandler object to parse the XML file that has the iTextSharp XML elements
xmlHandler.Parse("ExampleDoc.xml");

Of course there's more to this because the XML document must be in a format that iTextSharp can use. Everything is in the itext.dtd file in the example, but here is a summary:

The outermost element is itext and one can have title, subject, keywords and author attributes
Within that you can have one of the following elements:
- chunk - a small piece of text - useful if you want to set the colour or background of the text
- phrase - a collection of chunks and allows you to define the space between lines
- paragraph - a collection of chunks/phrases with a carriage return after them
The list and listitem element is used to add lists
You can add images to paragraphs, listitem and several other elements using the image element and url attribute
Tables are controlled through table, row and cell elements

Here is an example XML file:

This is what the pdf file generated using the xml above looks like: iTextXmlToPdf Example.pdf

Open Source Accounting Packages

Sat, 16 Jul 2005 12:27:38 GMT

At work we've been considering writing our own accounting package, so I did a little investigation into open source accounting packages. This is what I found:

Grisbi - This is a personal financial management package but it seems pretty good. The main advantage I see in a package like this is the fact that it has been kept simple. It supports budgets, financial years, multiple accounts, currencies and users. However it sounds like it does not do double entry accounting which is very strange.
GnuCash - It alows you to track bank accounts, stocks, income and expenses. As quick and intuitive to use as a checkbook register, it is based on professional accounting principles to ensure balanced books and accurate reports. The fact that it does not run on windows is a major problem.
Lazy8 Ledger - This accounting program is designed for people who have experience in bookkeeping.
MyPhpMoney - This is a basic personal financial management system
WebERP - This is a PHP based ERP/accounting system. It contains the standard accounting features one would expect. The interface is really basic and not really user friendly. For example when capturing a sales order one has to remember the part code or go to another screen to do a parts search.
OpenAccounting - A PHP accounting and inventory system. Features are: complex taxation handling, multi-currency support, sales, purchase orders, inventory and multi-user configuration. The demo site was pretty broken which is not a good sign.
XIWA - A double entry accounting system that allows you to import Quickbooks files. It supports multiple ledgers, sub entities or departments within ledgers and does payroll. It is written in Perl and uses a postgreSQL backend database.
SqlLedger - SQL-Ledger is a double entry accounting system. Accounting data is stored in a SQL Server. The entire system is linked through a chart of accounts. Each item in inventory is linked to income, expense, inventory and tax accounts. When items are sold and purchased the accounts are automatically updated. The web frontend is written in Perl.
Open Business Accounting - OBA is a web based business accounting software modified from sql-ledger to suit accountants ,auditors and who required massive data entry.
Compiere - Another ERP/accounting system. It is a Java based application and not web based.
Open For Business - A Java based ERP system. This one may was the most interesting of all the applications I looked at. If I had to start my own company I may very well use this.

Another list of free accounting packages can be found here.

None of these really seems to fit the bill for what we are looking for. Oh well.