Comments on: EC2, Confluence, S3 and PostgreSQL http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/ Notes on things I'm thinking and doing Thu, 04 Dec 2008 01:34:13 +0000 http://wordpress.org/?v=2.5.1 By: Ravi http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-7973 Ravi Fri, 28 Nov 2008 19:35:40 +0000 http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-7973 <p>I am a bit undecided on EBS,</p> <p>EBS seems to me like another VM with its instance store exposed as an iSCCI block device, but you pay for the allocation regardless of the usage (0.10 / gb / month). The reason I say this is that while data may persist- Amazon says that at any point in time it could fail (AFAIK) ... so you would still need to make incremental snapshots out to S3.</p> <p>I guess EBS is allows you to pull the logic of keeping data consistent etc. away from machine instances, also EBS blocks cannot be mounted by many instances @ once, I guess the primary instance could export it as nfs.</p> I am a bit undecided on EBS,

EBS seems to me like another VM with its instance store exposed as an iSCCI block device, but you pay for the allocation regardless of the usage (0.10 / gb / month). The reason I say this is that while data may persist- Amazon says that at any point in time it could fail (AFAIK) … so you would still need to make incremental snapshots out to S3.

I guess EBS is allows you to pull the logic of keeping data consistent etc. away from machine instances, also EBS blocks cannot be mounted by many instances @ once, I guess the primary instance could export it as nfs.

]]> By: Tom Davies http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-7083 Tom Davies Sun, 02 Nov 2008 04:33:17 +0000 http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-7083 <p>@Patrick -- Yes, EBS makes this sort of complex solution redundant.</p> @Patrick — Yes, EBS makes this sort of complex solution redundant.

]]> By: Patrick http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-7072 Patrick Sat, 01 Nov 2008 23:01:59 +0000 http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-7072 <p>A much better idea would be to mount the S3 as a drive on your EC2 instance and point the data directory of postgres to that mount point. Voila! No hassle consistent database.</p> <p>This how to shows you how to do it with MySQL - postgerSQL is basically the same method.</p> <p>http://www.sunsetlakesoftware.com/2008/09/13/running-drupal-website-amazon-ec2</p> A much better idea would be to mount the S3 as a drive on your EC2 instance and point the data directory of postgres to that mount point. Voila! No hassle consistent database.

This how to shows you how to do it with MySQL - postgerSQL is basically the same method.

http://www.sunsetlakesoftware.com/2008/09/13/running-drupal-website-amazon-ec2

]]> By: Tom Davies http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-1559 Tom Davies Sat, 12 Apr 2008 11:12:02 +0000 http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-1559 <p>PS: I'm not familiar with MySQL NDBIO...</p> PS: I’m not familiar with MySQL NDBIO…

]]> By: Tom Davies http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-1558 Tom Davies Sat, 12 Apr 2008 11:11:01 +0000 http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-1558 <p>I never installed a complete setup on an EC2 node, although I did test Postgres backups to S3 from my computer. I'm happy that Postgres backup/restore to S3 is practical.</p> <p>I didn't set up a Confluence cluster, but EC2 does allow UDP, so Confluence clustering will work.</p> <p>I would recommend that you use the largest machine EC2 provides before resorting to clustering, which introduces some overhead.</p> <p>I don't now what the worst case UDP latency between two EC2 nodes is -- you would need to test this (or ask Amazon) before creating a cluster.</p> I never installed a complete setup on an EC2 node, although I did test Postgres backups to S3 from my computer. I’m happy that Postgres backup/restore to S3 is practical.

I didn’t set up a Confluence cluster, but EC2 does allow UDP, so Confluence clustering will work.

I would recommend that you use the largest machine EC2 provides before resorting to clustering, which introduces some overhead.

I don’t now what the worst case UDP latency between two EC2 nodes is — you would need to test this (or ask Amazon) before creating a cluster.

]]> By: Matthew Arrott http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-1544 Matthew Arrott Fri, 11 Apr 2008 21:21:08 +0000 http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-1544 <p>Did you run multiple instance of Conflunence as a fault-tolerant cluster? Did you consider running a MySQL NDBIO cluster for the Fault-Tolerant DB?</p> <p>Great to see that some has precede us. Many thanks,</p> <p>Matthew</p> Did you run multiple instance of Conflunence as a fault-tolerant cluster? Did you consider running a MySQL NDBIO cluster for the Fault-Tolerant DB?

Great to see that some has precede us. Many thanks,

Matthew

]]> By: Ram http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-1253 Ram Tue, 18 Mar 2008 16:47:42 +0000 http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-1253 <p>I would be interested in updates too :)</p> I would be interested in updates too :)

]]> By: Adam Sherman http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-778 Adam Sherman Thu, 14 Feb 2008 18:54:50 +0000 http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-778 <p>Any news on this project? I'm working on a project now that is PostgreSQL based and needs to be scaled.</p> <p>A.</p> Any news on this project? I’m working on a project now that is PostgreSQL based and needs to be scaled.

A.

]]> By: Tom Davies http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-505 Tom Davies Sun, 13 Jan 2008 09:16:30 +0000 http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-505 <p>Thanks for the link, Scott. Interesting that they have 6 months uptime for their DB server.</p> Thanks for the link, Scott. Interesting that they have 6 months uptime for their DB server.

]]> By: Scott Royston http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-484 Scott Royston Fri, 11 Jan 2008 02:42:24 +0000 http://diversions.nfshost.com/blog/2007/12/22/ec2-confluence-s3-and-postgresql/#comment-484 <p>I'm very interested in hearing how this goes. From poking around google it looks like 'GoPlan' is already doing it. http://comments.deasil.com/2007/05/24/goplan-interview-ec2/</p> I’m very interested in hearing how this goes. From poking around google it looks like ‘GoPlan’ is already doing it. http://comments.deasil.com/2007/05/24/goplan-interview-ec2/

]]>