Jump to content

Suggestion: CDN to speed-up downloads and reduce ping

Fel1x

By Fel1x
in Technical Issues

 Share

Recommended Posts

JamesP Newbie

JamesP

Posted
Posted

We were getting totally slammed last night and the server was barely accessible. We put a rate limit on the downloads so that people are still able to access them, but that unfortunately means that people will be slowed down.

 

We are looking into other alternatives.  If you have any specific cost-effective suggestions, I would be happy to hear them.

Link to comment
Share on other sites
Wesley Newbie

Wesley

Posted
Posted

For starters, I will put a link to the torrent thread for any large number of downloads or more regular users:

http://board.freedomainradio.com/topic/34236-more-torrents/?hl=torrent#entry317261

 

Besides that, I think we may need information as to bandwidth usage and what cost is considered low cost. Doing a quick Google search yields sites with various tiered plans:

http://www.maxcdn.com/pricing/ (Monthly, click button for yearly)

https://www.cloudflare.com/business (bottom of page, start at $200, though they likely get a bit higher with higher requirements)

http://aws.amazon.com/cloudfront/ (Only pay for number of requests and data transferred)

 

This is just from searching CDN quick on Google, so it is easily possible that none of these are good options or that there are better options than CDNs available.

Link to comment
Share on other sites
JamesP Newbie

JamesP

Posted
Posted

Thanks for the links. Can you make any recommendations per your experience? I'm pretty new to CDNs from the tech side of things.

 

I can get more information regarding the throughput expectations later. But just for now, the traffic to the site had increased such that the 100mbps network interface was completely slammed. There are no available upgrades with the current hosting provider.

 

The other thing to keep in mind is that we do have a bandwidth allowance of 20TB per month on the webserver so anything that allows us to take advantage of that would be a plus, but it's not absolutely required.

We also have plenty of application capacity; it's the network that limits us right now.

Link to comment
Share on other sites
Kevin Beal Newbie

Kevin Beal

Posted
Posted

We use cloudfront and Amazon S3 at the company I work at. We don't get 20TB / month of bandwidth by any means, but we chose that solution precisely because it is so scalable. And it's an awesome service using cloudfront because it's in the cloud, cached, quick etc. Their S3 pricing table looks like this:(Standard Storage  vs  Reduced Redundancy Storage  vs  Glacier Storage)

First 1 TB / month$0.095 per GB$0.076 per GB$0.010 per GBNext 49 TB / month$0.080 per GB$0.064 per GB$0.010 per GBNext 450 TB / month$0.070 per GB$0.056 per GB$0.010 per GBNext 500 TB / month$0.065 per GB$0.052 per GB$0.010 per GBNext 4000 TB / month$0.060 per GB$0.048 per GB$0.010 per GBOver 5000 TB / month$0.055 per GB$0.037 per GB$0.010 per GB Cloudfront's pricing is very similar I believe

Link to comment
Share on other sites
Wesley Newbie

Wesley

Posted
Posted

Maybe I am confused, but wouldn't using a Content Delivery Network have the content on multiple servers which would each have their own network interface speed? Thus, it would be something like 100 mbps for each distinct server. Maybe I am completely misunderstanding.

 

Short Overview of CDN costs/benefits:

http://whatismyipaddress.com/cdn

 

Many of the older podcasts may not need the boost in download speed, or the torrents could be encouraged. However copying the newest/ most trafficked podcasts to many servers to download from would improve latency and would make it so you wouldn't run into the network speed cap.

 

If someone has better knowledge to weigh in on this, please do.

Link to comment
Share on other sites
Kevin Beal Newbie

Kevin Beal

Posted
Posted

Maybe I am confused, but wouldn't using a Content Delivery Network have the content on multiple servers which would each have their own network interface speed? Thus, it would be something like 100 mbps for each distinct server. Maybe I am completely misunderstanding.

 

Short Overview of CDN costs/benefits:

http://whatismyipaddress.com/cdn

 

Many of the older podcasts may not need the boost in download speed, or the torrents could be encouraged. However copying the newest/ most trafficked podcasts to many servers to download from would improve latency and would make it so you wouldn't run into the network speed cap.

 

If someone has better knowledge to weigh in on this, please do.

 

You also benefit from getting data from a closer server farm. If one server goes down, it's still up and available. They often cache that kind of thing so it's in memory and don't have to read from disk. A whole lot of benefits that I've never heard about, I'm sure.

 

A lot of people use CDN's when they want to reference CSS and Javascript libraries that lots of websites have (Bootstrap or jQuery for example) so that it's cached in the browser the first time you visit a website that requires that resource, and you don't have to download it any of the next times you visit a page that uses it.

 

CDN's are all about uptime so that when you separate your app server from a huge list of files on a CDN, your app server can go down and all your files are still up.

Also, they are built to scale so that you don't have to worry about huge numbers of visitors (except in how it affects your wallet).

Link to comment
Share on other sites
JamesP Newbie

JamesP

Posted
Posted

So yesterday we had 500GB of bandwidth consumption. That would include us getting swamped and having to rate limit during the last few hours of the evening.  I can only suspect that today would be similar.

 

We don't have 500GB days every day, but it's reasonable to expect that we will be seeing more of them in the future.  We're averaging around 250GB per day right now.

 

Yay growth! :)

Link to comment
Share on other sites
Fel1x Newbie

Fel1x

Posted
  • Author
Posted

Here's a good overview of why latency matters and why a CDN is a good idea. The more time you spend transmitting to client the longer the queue gets. Link to original text.http://blog.cachefly.com/2012/02/15/understanding-traceroute/#.UkKHFqdRWId

Queuing Delay To understand queuing delays, first you must understand the nature of interface utilization. For example, a 1GE port may be said to be “50% utilized” when doing 500Mbps, but what this actually means is 50% utilized over some period of time (for example, over 1 second). At any given instant, an interface can only be either transmitting (100% utilized), or not transmitting (0% utilized). When a packet is routed to an interface but that interface is currently in use, the packet must be queued. Queuing is a natural function of all routers, and normally contributes very little latency to the overall forwarding process. But as an interface approaches the point of saturation, the percentage of packets which must be queued for significant periods of time increases exponentially. The amount of queuing delay that can be caused by a congested interface depends on the type of router. A large carrier-class router typically has a significant amount of packet buffering, and can add many hundreds or thousands of milliseconds of latency when routing over a congested interface. In comparison, many enterprise-class devices (such as Layer 3 switches) typically have very small buffers, and may simply drop packets when congested, without ever causing a significant increase inlatency.Matt: Once you’ve read this once or twice, it makes all the sense in the world.. But did you ever really look at a gige port doing 512kbps and think of it as operating at 100% utilization?

Link to comment
Share on other sites
Freedomain Contributor

Freedomain

Posted
Posted

Howdy everybody, thanks for the feedback and ideas about how to solve this problem.  We're going to invest in a $600/month CDN package to solve this issue and allow us to scale properly for any future growth.

 

It's a significant monthly bill, but it's an important investment that is unavoidable at this point. Apparently we did 750 GB of downloads in the last 24 hours, which is absolutely insane.

 

We should hopefully have everything set up in the next few days. Fingers crossed.

 

If anybody would like to chip in to help offset some of these costs, that would certainly be appreciated.

 

http://www.freedomainradio.com/donate.aspx

Link to comment
Share on other sites
fer Newbie

fer

Posted
Posted (edited)

OK, I figured out what happened, sorry about that! That file was omitted in the upload because the filename didn't match the pattern I was using.  It should be available in about an hour.

 

http://media.freedomainradio.com/feed//handbook_human_ownership/FDR_Book_the_handbook_human_ownership.mp3

and

 

http://media.freedomainradio.com/feed//against_the_gods/FDR_Book_against_the_gods.mp3

are downloading really slowly, heads up if they aren't on the cdn yet.

Edited by fer
Link to comment
Share on other sites
  • 7 months later...
Fel1x Newbie

Fel1x

Posted
  • Author
Posted

I listened to a recent podcast where there was talk about CDN-package overages. Since this is a growing show I might suggest a Pay As You Go CDN like CDN.net which is quite flexible, because you can choose your locations depending on your needs, also you won't incur overage charges. I don't have experience with CDN.net, but there references seem legit.

Link to comment
Share on other sites
Freedomain Contributor

Freedomain

Posted
Posted

We're very happy with our current CDN and it's WAY more cost effective than CDN.net.

 

We purchased a certain package which is good for 12 months, but we're going to run through the allotted bandwidth months early.

 

It's a good problem to have, but it's an increase in expenses that we wanted to communicate.

Link to comment
Share on other sites
 Share
Go to topic listing
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.

  I accept