.NET and MongoDB
Building Applications with NoRM and MongoDB
New York ALT.NET Group Meetup
2010-07-28
John C. Zablocki
Development Lead, MagazineRadar
Adjunct, Fairfield University
Agenda
What is NoSQL?
NoSQL Databases
Introducing MongoDB
.NET Drivers for MongoDB
Introducing NoRM
Case Study: RateMySnippet.com
Questions?
NoSQL
What is NoSQL?
Coined in 1998 by Carlos Strozzi to describe a
database that did not expose a SQL interface
In 2008, Eric Evans reintroduced the term to
describe the growing non-RDBMS movement
Broadly refers to a set of data stores that do not
use SQL or a relational model to store data
Popularized by large web sites such as Google,
Facebook and Digg
NoSQL Databases
Wide Column,
Key/Value
and Document Store...
Oh My!
NoSQL Databases
NoSQL databases come in a variety of flavors
XML (myXMLDB, Tamino, Sedna)
Wide Column (Cassandra, Hbase, Big Table)
Key/Value (Redis, Memcached with
BerkleyDB)
Object (db4o, JADE)
Graph (neo4j, InfoGrid)
Document store (CouchDB, MongoDB)
Why NoSQL?
RDBMSs and the Environment
RDBMS Administrators are highly paid
Highly paid individuals often buy larger than
average homes or cars
Larger than average homes and cars require
more energy than smaller home and cars
Therefore RDMBSs contribute to global
warming more than NoSQL databases which
typically do not require the addition of a DBA
RDBMSs and the Environment
RDBMSs often require high end servers and
that are taxing on disks
High end servers consume more electricity than
mid-range servers
Taxed disks fail more often than untaxed disks
Therefore RDBMSs require more energy and
produce more waste (lots of hard drives in
landfills) than NoSQL DBs, which run on mid-
range servers.
Why NoSQL?
NoSQL and Healthcare
The current healthcare crisis requires talented
software engineers to fix the outdated or non-
existent IT systems of the hospital system
Talented software engineers spend a great deal
of time mapping objects to tables in RDBMSs
Talented software engineers are unable to fix
healthcare because they are mapping objects to
tables
Therefore RDBMSs are causing illnessnes
Dropping ACID
Dropping ACID
CAP Theorem and Eventual Consistency
A distributed computer system cannot provide
Consistency, Availability and Partition
Tolerance at the same time
Immediate consistency is expensive
Some systems can tolerate dirty reads
Removing relational storage structure helps to
reduce transactional requirements
Inserts/Updates written to fewer places
Dropping ACID
ACID support is not required for blogs, status
updates, product listings, etc.
Would a 2 second delay in finding out who the
mayor of Turnpike Spirits is matter?
Or that your friend is embarrassed to admit
that she loves Jersey Shore?
Modern relational databases simply do not
scale to Internet proportions
Sites like Digg, Facebook and Ebay have data
sets 10s or 100s of TB or even PB large
NoSchema Databases
NoSQL databases generally do not impose a
formal schema
Optimized storage for unstructured data
NoSchema Documents
Related data is stored in a single document
The ubiquitous Blog example
Each post is a document
Post might have reference to Blog or User
document or redundantly store blog/user data
East post document has a nested collection of
tags and comments
Application layer must be smarter about
referential integrity
NoSchema Documents
var post = {
Author : { Name : "John Zablocki", _id : "497ce96f..." },
Title : "On Running NerdDinner on MongoDB with NoRM",
Tags : ["mvc", "mongodb", "norm"],
CreatedDate : new Date('6-14-2010'),
Text : "At the Hartford Code Camp..."
}
NoSchema Documents
John Zablocki
497ce96f395f2f052a494fd4
On Running NerdDinner on MongoDB - Part 1
mvc
mongodb
norm
6-14-2010
At the Hartford Code Camp...
Something Completely Different
Introducing MongoDB
Open source, document-oriented database
10gen corporate entity behind development
10gen supports official drivers for many
platforms, but not .NET!
MongoDB – The Basics
Schema-less documents stored in collections
Documents are stored as BSON (Binary JSON)
JavaScript used to query and manipulate
documents and collections
Each document in a collection has a unique
BSON ObjectId field named _id
Collections belong to a database
MongoDB – Server Features
Indexes
Secondary, compound and unique
Replication
Master/slave and replica sets
Auto-sharding (1.6)
MapReduce
GridFS
Large file storage
MongoDB – Hidden Features
MongoDB will help you:
Make more friends
Be more confident in social situations
Be more attractive to members of the opposite
(or same) sex
MongoDB on Windows
Installing MongoDB on Windows
Download the binaries from mongodb.org
Extract to Program Files directory (or wherever)
Create a directory c:\data\db
Run mongod.exe from the command line with the --
install switch
See http://bit.ly/aed1RW for some gotchas
To run the daemon without installing, simply run
mongod.exe without arguments
Run mongo.exe to verify the daemon is running
MongoDB - Shell
The MongoDB interactive JavaScript shell
(mongo.exe) is a command line utility for
working with MongoDB servers
Allows for CRUD operations on collections
May be used for basic administration
Creating indexes
Cloning databases
Also useful as a test-bed while building apps
MongoDB - Shell
/*Connect to a server:port/database
(defaults are localhost:27017/test ):*/
mongo.exe localhost:27017/AltNetGroup
//Switch database:
use CodeCamp
//View collections in a database:
show collections
//create an index on Name field
db.Artists.ensureIndex({ Name : 1 });
//copy one database to another
db.copyDatabase("CodeCamp", "AltNetGroup")
MongoDB – Shell CRUD
//Insert an item into a collection
db.Artists.insert({ Name : “The Shins” });
//Find an item in a collection:
db.Artists.findOne({ Name: “Radiohead”});
//Find items in a collection:
db.Artists.find({ Name : /The/i});
//Count items in a collection
db.Artists.count();
MongoDB – Shell CRUD
//Update an item in a collection
db.Artists.update({ Name : “JackJohnson” },
$set : { Name : “Jack Johnson” } });
/*Update items in a collection
Without $set in this example, whole
document would be replaced!*/
db.Artists.update({ Name : { $ne : null } },
{ $set : { Category : “Rock” } }, false, true);
/*$ denotes special operators and operations
$push, $pop, $pull, etc.*/
MongoDB – MapReduce
MapReduce is used for batch manipulation and
aggregation of data
A Map function transforms a collection into a
series of key/value pairs
A Reduce function operates on that key/value
pair series to produce output based on those
input values
An optional Finalize function is called after the
reduce function is invoked
MongoDB – Shell MapReduce
//Create some data
db.Posts.insert(
{
Name : "On Installing MongoDB as a Service On Windows",
Tags : ["mongodb"]
});
db.Posts.insert(
{ Name : "On Running NerdDinner on MongoDB with NoRM",
Tags : ["mongodb", "norm", "mvc"]
});
db.Posts.insert(
{ Name : "On A Simple IronPython Route Mapper for ASP.NET MVC",
Tags : ["mvc", "ironpthon"]
});
MongoDB – Shell MapReduce
/*
Create the map function
This function creates a view that looks like
{ “mvc”, [1, 1] },
{ “norm”, [1] }
*/
var map = function() {
if (!this.Tags) { return; }
for (var index in this.Tags) {
emit(this.Tags[index], 1);
}
};
MongoDB – Shell MapReduce
/*
Create the reduce function
conceptually, reduce gets called like:
reduce("mvc", [1, 1]);
reduce("norm", [1]
/*
var reduce = function(key, vals) {
var count = 0;
for (var index in vals) {
count += vals[index];
}
return count;
};
MongoDB – Shell MapReduce
/*
Run the mapreduce command on the Posts collection
using the map and reduce functions defined above
store the results in a collection named Tags
*/
var result = db.runCommand(
{
mapreduce : "Posts",
map : map,
reduce : reduce,
out : "Tags"
});
db.Tags.find()
.NET and MongoDB
.NET and MongoDB
No officially supported 10gen driver
mongo-csharp
Actively being developed and supported
Supports typed collections
Simple-mongodb
Json-centric
Actively developed, but less so than other
drivers like mongo-csharp and NoRM
NoRM
NoRM
OK, it is Pronounced No R M!
Actively developed by Andrew Theken and
James Avery, among others
Active community with reliable support
I received help even as I prepared this slide!
Support for typed and untyped collections,
MapReduce, Property Mapping, LINQ,
Expando, nearly all CRUD operations
NoRM – The Basics
//Connections managed with IDisposable pattern
//Connect to test database on localhost
using (Mongo mongo = Mongo.Create("mongodb://localhost/test")) {
Console.WriteLine(mongo.Database.DatabaseName);
}
//Mongo instance has MongoDatabase property
/*MongoDatabase has GetCollection methods for accessing MongoCollection
instances*/
//CRUD operations performed on collections
NoRM - CRUD
//Class properties map to document structure
public class Artist {
//ObjectId property mapped to _id field in document
public ObjectId Id { get; set; }
public string Name { get; set; }
//IList property will be mapped to JavaScript Array
private IList _albums = new List(0);
public IList Albums {
get { return _albums ; }
set { _albums = value; }
}
}
NoRM - CRUD
var artist = new Artist() { Name = "BadLands" };
//Inserting a document into a typed collection
mongo.Database.GetCollection("Artists").Insert(artist);
//Updating (replacing) a document in a typed collection
artist.Name = "Badlands";
mongo.Database.GetCollection("Artists").Save(artist);
//Updating a nested collection
mongo.Database.GetCollection("Artists").UpdateOne(
new { Name = "Badlands"},
new { Albums = M.Push("Voodoo Highway") }
);
NoRM - CRUD
//Find all documents in a typed collection
var artists = mongo.GetCollection("Artists").Find();
Console.WriteLine(artists.FirstOrDefault().Name);
//Query with a document spec
var artist = mongo.GetCollection("Artists")
.FindOne(new { Name = "Badlands"
});
Console.WriteLine(artist.Albums.Count);
//Count the documents in a collection
long count = mongo.GetCollection("Artists").Count();
Console.WriteLine(count);
NoRM - MapReduce
//Add a Tags collection added to Artist class
private IList _tags;
public IList Tags {
get { return _tags; }
set { _tags = value; }
}
//Add some tags – use Set not PushAll, since Tags was null
mongo.Database.GetCollection("Artists").UpdateOne(
new { Name = "Badlands" },
new { Tags = M.Set(new List()
{ "Hard Rock", "80s" }) });
NoRM - MapReduce
//Create map and reduce functons
string map = @"function() {
if (! this.Tags ) { return; }
for (index in this.Tags) {
emit(this.Tags[index], 1);
}
}";
string reduce = @"function(previous, current) {
var count = 0;
for (index in current) {
count += current[index];
}
return count;
}";
NoRM - MapReduce
//MapReduce class is responsible for calling mapreduce command
MapReduce mr = mongo.Database.CreateMapReduce();
//Represents the document passed to the
//db.runCommand in the shell example
MapReduceOptions options = new MapReduceOptions("Artists")
{
Map = map,
Reduce = reduce,
OutputCollectionName = "Tags"
};
MapReduceResponse response = mr.Execute(options);
var collection = mongo.Database.GetCollection("Tags");
Console.WriteLine(collection.Count());
NoRM - LINQ
//LINQ provider exposed via AsQueryable method of MongoCollection
var artists = mongo.Database.GetCollection("Artists")
.AsQueryable();
//Find items in typed collection
var artistsStartingWithB =
from a in mongo.Database.GetCollection().AsQueryable()
where a.Name.StartsWith("B")
select a;
Console.WriteLine("First artist starting with B: " +
artistsStartingWithB.First().Name);
NoRM - LINQ
//Find artists without pulling back nested collections
//Note use of Regex name search
var artistsWithoutAlbums =
from a in mongo.Database.GetCollection().AsQueryable()
where Regex.IsMatch(a.Name, "land", RegexOptions.IgnoreCase)
select new { Name = a.Name };
Console.WriteLine(artistsWithoutAlbums.First().Name);
//Find artists with a given tag
var artistsWithHardRockTag = mongo.Database.GetCollection("Artists")
.AsQueryable().Where(a => a.Tags.Any(s => s == "Hard Rock"));
Console.WriteLine(artistsWithHardRockTag.First().Name);
Case Study: RateMySnippet.com
Final Thoughts
Eat food. Not too much. Mostly plants.
- Michael Pollan
Final Thoughts
Write Code. Not too much. Mostly C#.
- John Zablocki
Final Thoughts – The Stack
ASP.NET MVC
MVCContrib
MongoDB
NoRM
SQL Server
NHibernate
FluentNHibernate
Spring.NET
Recoil
AutoMapper
Q&A
http://dllHell.net - my blog
http://www.CodeVoyeur.com - my code
http://www.linkedin.com/in/johnzablocki
http://twitter.com/codevoyeur
http://mongodb.org - Official MongoDB site
http://groups.google.com/group/norm-mongodb
http://www.summerofnorm.com - Coming Soon!