Very interesting project, I’d love to see this as an official EE module

I’m keen to start things moving on this plugin soon. While it will start it’s life as simply enabling uploads to Amazon S3 via the upload function.

The scope of this project for me is growing every day. While my initial objective for this plugin was to offload movie content to S3 for some of our client sites. The more I think about it I would like to replace a lot of the functionality present in the upload feature.

Things I really want to bring to the party are:

* upload progress monitoring;
* management of uploaded files (like deleting/renaming previously uploaded files);
* customise options based on weblog, so do away with Insert URL or Insert Image, and present only what option is required for specified weblog;
* possibly introduce some predefined css to apply to image tags, again probably weblog dependent.

The more and more I thought about it, the Amazon S3 part for us will ultimately become a feature of a much bigger module.

So if anyone has any more ideas along these lines, them chime in, as I want to get a developer onto this probably in the next month.

How much of my ideas are actually possible within the EE framework I’m unsure, it’s not my area, I’m just want the goodies :D

I thing I have been finding out lately is that the Amazon web services are actually pretty expensive when you are using a lot of bandwidth.

Not when you live in Australia! Our bandwidth prices over here are a killer, and when your performing offsite backups that really starts to add up.

So for us it’s a real winner.

It would come much further down the road for sure, but I was already thinking of allowing for uploads to other services, maybe by supporting FTP. In my case we could then hosting our site and DB in Aus, but allow for a lot of the larger content to be uploaded to servers with access to cheaper bandwidth.

Our cost’s for bandwidth, and more particularly storage over the last 18 months have gone through the roof, mainly due to the explosion on video content for our average client.

Are you using S3’s European datacenters?

Well I’m not really using either in any large scale just yet. But what little we are using is all the US DC’s. Latency from Australia to Europe is huge.

I guess it depends on the situation.  Smugmug says S3 saves them money, Matt Mullenweg said it doesn’t.

Yes was reading the Smugmug article yesterday.

So many variables. For us there are so many benefits, that even if the bandwidth did cost more than our local DC, the cost savings in other areas would win the day anyway.

In particular Amazon S3 would appear to be considerably more reliable than a single server solution. We have for some time been looking at ways of breaking services out of a single DC for greater redundancy.

For sure, that brings up a good point though.  You don’t want to stick all your eggs in one basket.  S3 can go down just like anything else.  I got caught in the BingoDisk/Strongspace downtime which lasted for over a week.

Oh we already have that covered, it’s really quite simple.

Data is still stored on the local server. Our monitoring server, it it detects S3 has gone down instantly updates the dns record to point back to our box again, which would result in less than 5 minutes of downtime.

Or alternatively you can do the same by mirroring it to the S3 European DC, ie having the DNS record updated automatically in the case of failure.

We offload all our MySQL to a cluster too which greatly reduces downtime.

Hiya,

I don’t meant to en-crouch on the thread or anything but I saw the S3 product quite a while ago and thought that the pricing was amazing. I just looked again and used their online calculator to test out some costs and they seem exceptionally good.

I went with the following variables on the European centres :

100GB of storage
10GB Transfer in per month
20GB transfer out per month
50,000 PUT/LIST requests &
20,000 other requests

This came out to a total of $23.22 or just under £12 in my currency. I personally would have difficulty here in the UK finding a reliable host that could even offer me the 100GB of monthly storage at that price! I’m not sure if I am understanding that part correctly but if I am then I really see it as being a fantastic cost saver. John could you possibly explain a bit more why you think it comes out costly as when I used the calculator it in fact seems like a really sweet deal although I will be the first to admit that I didn’t totally understand what it meant because if you say you have 100GB of data stored and that you are then adding 10GB per month then how can they calculate it as you would then have 110GB - 120GB - 130GB and so on and so forth.

As John says though there is always downtime to contend with but is there really anywhere in the world that can offer you a 100% uptime SLA. I don’t think anyone could ever do that and if they did then I would personally have to call them a liar as all data centres could go down at any time. If, god forbid a satellite fell out of the sky onto a data-centre and crushed all the servers then there’s not really much they could do about it is there?

Amazon on the other hand have millions if not billions of capital behind them so I would expect their services to be as good as, if not better than most. You are paying for the service after all.

If my understanding is wrong on any of this then please do feel free to put me right as I think that from what I have seen so far the S3 service sounds really really attractive.

Thanks for any help with understanding all of this more!

Best wishes,

Mark

John was pointing out in particular the cost of bandwidth. I think the bandwidth cost can be seen to be costly if your used to buying American bandwidth, which sells for a silly prices. Here in Australia you can’t get it for anywhere near as cheap.

But for me there is so much more to it aside from moving that data about. Once I factor all our other cost savings, of which there are many, it blows any other solution out of the water for us. I mean by offloading a good 100gb of data from just one server, I’m not just saving on storage costs, I’m saving on backups, hardware maintenance and upgrades, server load. Not to mention when a server goes down I have far less headaches.

And as S3 has the European DC, I can mirror between the two. Sure Amazon could go bust… but it’s a pretty big maybe. If that day comes I will not be silly enough to not have a backup.

I should also point out that the reason I was initially interested in S3 was for backup purposes anyway, for which data transfer costs are largely not an issue. We are slowly transferring an amount of our archives onto S3 as part of a disaster recovery plan.

Our own testing, and other results I have seen seem to point to it being exceptionally reliable.

Mark, do you know if the cost of data in Europe is much higher than the US? I’m curios as we are probably moving back to the UK at some point in the near future, so it’s good to know!

Hope that clears it up a little for you.

Hiya,

Thanks for the reply. Well I just put the same figures into the calculator and it came back with :

Amazon S3 (US) - $20.12
Amazon S3 (EUR) - $23.22

So not really a lot in it. How cheap is bandwidth in the US then. Over here it is quite cheap but I don’t think as cheap as Amazon are offering. I know that over here though most places just say that you get so much bandwidth per month and don’t actually charge you any more for it unless you go over your allowance but it can become costly if you do go over it as most places, unless you go totally dedicated don’t really offer massive amounts.

Where are you going to be based in the UK if you do move back again? Are you originally from the UK then and if so where abouts?

Best wishes,

Mark

I don’t have any dedicated hardware in the US, so have never really had quotes on US data. But to compare, I just randomly checked the price of a few dedicated boxes. One box had 2000GB of data to serve 200GB of storage for $250 US. So that gives you an idea of how stupidly cheap bandwidth is in the US. Here for the same price I get 100GB of bandwidth! So that’s why in my case at least the two just don’t compare.

It’s one thing to buy cheap data, it’s quite another to have good quality hardware on the other end of the pipe.

But also in my case, as I’m not moving masses of data, I’m paying for data I’m not using. I think last month we only used half our quota… yet if I go over my quota I’m charge AUD$3 a gb!

I think that’s the other advantage for people like me, Amazon S3 only changes you for what you use. With no minimum too. I think the first month I used it, they debited my card for $0.17!! In situations where your not consuming large amounts of bandwidth, this really starts to add up. I think this is the guts of what we came to the conclusion of earlier, it all depends on your on unique situation.

As for me and the UK. Yes born there and moved out here 8 years ago. So will be back to my home town again, Barnstaple, Devon. That’s the joys of this business, it’s very portable Just came back after spending Christmas there, that was enough to swing it!

BTW… doozazzle

Another great bio reader!!

On a side-note to all of this would you have any ideas of how easy it would be to set up Amazon S3 with EE so that instead of the whole site being on their servers you could just have file downloads stored on their and then reference them inside of EE so that you could offload storage in that way.

At the moment the server that I am hosted on is pretty good but storage does start to get quite expensive so I am looking at a way of storing the data somewhere else and then allowing people to download files from there instead.

I also use LinkLok URL from http://www.vibralogix.com to hide the download links to the files and secure them in that way. Would it be possible to get all of this somehow working together as one?

Thanks for any help on that.

Best wishes,

Mark

Mark, I offer a least one download that is stored on S3, and many of my images. I have a Firefox addon that helps with that considerably.

Hi Sue,

So how easy is it to program this kind of thing into EE then? Care to share the link to the download on Amazon or is it for something private? Is it one of your entries on EE How To perhaps?

Thanks.

Best wishes,

Mark

FFFA - Amazon S3 Firefox Organizer

Added just a few seconds ago just for you.

(Part of my attempting to have new topics every day.)