FriendlyHacker Posted April 29, 2014 Posted April 29, 2014 Am very busy with the incoming Joomla day event next weekend, but can definetly write a bot to download every single video in the coming weeks, so anyone with enough hard drive space can click the go button and have it downloaded locally. I would think this needs to be done by a small amount of people though, we don't want to risk causing trouble to google if thousands of people run this software at the same time.
rothbard Posted April 29, 2014 Posted April 29, 2014 Does Stefan have all of the original pre-uploaded videos? Can we put them up on a private/public torrent tracker?
FriendlyHacker Posted April 29, 2014 Author Posted April 29, 2014 If you have the files locally, yes you can make as many torrent trackers you want, but first you need to download them, which is a laborious slow process that should be automated, heck even the video descriptions and comments can be saved.
Lians Posted April 29, 2014 Posted April 29, 2014 You can write a python script that does the job in less than a hundred lines of code using gdata.youtube.com and pytube but I wouldn't post anything publicly since downloading videos from YouTube is a violation of their ToS. https://www.youtube.com/static?gl=GB&template=terms you agree not to access Content or any reason other than your personal, non-commercial use solely as intended through and permitted by the normal functionality of the Service, and solely for Streaming. "Streaming" means a contemporaneous digital transmission of the material by YouTube via the Internet to a user operated Internet enabled device in such a manner that the data is intended for real-time viewing and not intended to be downloaded (either permanently or temporarily), copied, stored, or redistributed by the user.
rothbard Posted April 29, 2014 Posted April 29, 2014 If people are willing to donate money to me, I will rip the videos in HD, package them in a torrent, host a tracker, and seed the torrents.
FriendlyHacker Posted April 29, 2014 Author Posted April 29, 2014 You can write a python script that does the job in less than a hundred lines of code using gdata.youtube.com and pytube but I wouldn't post anything publicly since downloading videos from YouTube is a violation of their ToS. https://www.youtube.com/static?gl=GB&template=terms Exactly.
ParaSait Posted April 29, 2014 Posted April 29, 2014 Another thing I was thinking of is we could make mirrors on alternate video sites, like dailymotion and vimeo. This is of course a more work intensive process, maybe we can have someone do this for a fee. As a bonus side effect this is also probably a way to reach more people.
Wesley Posted April 29, 2014 Posted April 29, 2014 Ah, there is another thread about this. How much money in donations would be needed for this to happen? Also, I was wondering how valuable the entire video database would be and whether (as a backup) we might want to do somewhat lower quality in order for the torrent to be able to proliferate further and not be prohibitive for a seed to participate.
Lians Posted April 29, 2014 Posted April 29, 2014 I wrote a quick script today and calculated that ripping all 1533 FDR YouTube videos in maximum quality would require 383.6 GB of space. Downloading all the files would take approximately 8 hours and 31 minutes on my 100 Mbps connection.
Wesley Posted April 29, 2014 Posted April 29, 2014 I wrote a quick script today and calculated that ripping all 1533 FDR YouTube videos in maximum quality would require 383.6 GB of space. Downloading all the files would take approximately 8 hours and 31 minutes on my 100 Mbps connection. Wow, not as bad as I thought. I probably have that available on my storage drive, especially since I was considering getting NAS sometime soon anyway. I could probably be a seed for at least a couple days once someone got the torrent up.
Lians Posted April 29, 2014 Posted April 29, 2014 Wow, not as bad as I thought. I probably have that available on my storage drive, especially since I was considering getting NAS sometime soon anyway. I could probably be a seed for at least a couple days once someone got the torrent up. Torrents are always an option but I'm not sure how many people would be interested in them. I wrote the script to make it easy for multiple people to create backups in case YouTube decides to permanently suspend Stef's account. Over the next few days I'll create a user interface for non-techies, add an option to download the available metadata (thumbnails and descriptions) and clean-up the code.
FriendlyHacker Posted April 30, 2014 Author Posted April 30, 2014 Torrents are always an option but I'm not sure how many people would be interested in them. I wrote the script to make it easy for multiple people to create backups in case YouTube decides to permanently suspend Stef's account. Over the next few days I'll create a user interface for non-techies, add an option to download the available metadata (thumbnails and descriptions) and clean-up the code. Better not release that to the wild, in the wrong hands its a weapon to take the channel offline for good, it might seen like a DDOS too and that is known to end people in jail for 5 years.
Lians Posted April 30, 2014 Posted April 30, 2014 Better not release that to the wild, in the wrong hands its a weapon to take the channel offline for good, it might seen like a DDOS too and that is known to end people in jail for 5 years. Yeah, that's why I was going to release it privately. I'm essentially writing a web scraper. It's not that easy to DDoS Google's servers though. While testing the script I was getting 403 Forbidden errors whenever I created too many parallel requests. I managed to route the connections through the Tor network and I'm now trying to spawn multiple Tor processes with different IP's.
BlackHeron Posted April 30, 2014 Posted April 30, 2014 Google's (at least in lipservice) has been pretty good about standing behind its "data liberation" policy and allowing users to be able to download and control all their data stored on their servers. The issue has always been that a lot of the tools tend to break when you try to use them, especially if there is a LOT of data to download. I've tried a few times and failed with my own personal data and I've never evenn tried to retrieve my videos on YouTube. I think if someone with administrative privileges from the FDR google account tried to download all of the videos they couldn't say it was a DDoS attack although the sheer mass of videos involved would probably be unprecedented as a retrieval all at once. The process could take months officially and be a real headache for the FDR team. I think that doing it on the sly as Lians is talking about is probably the best way, and it doesn't require bothering Stefan, James or anyone else official from FDR to get this done, or taxing the server with a ton of bandwidth. It's a good thing you are doing though Lians. I have started torrenting a few of the podcast files and storing them on my computer to start helping seed them but there doesn't seem to much call for them. Not many people use torrents for legitimate uses any more. People have been scared off by the IP trolls. It's too bad as it is a good/cheap way to distribute large files in a decentralized way. I see a couple other peers out there seeding these but in the past 24 hours I've not gotten even the slightest hint of anyone biting on them to torrent. That's really killing my ratio on my torrent client but luckily I've got a bunch of Ubuntu/Xubuntu iso files seeding at the moment that are really hopping along and in high demand, so I think I'll be back up in positive ratio territory quite soon.
FriendlyHacker Posted April 30, 2014 Author Posted April 30, 2014 I would advise more patience and less parallel requests, this has to be done only once so no need to rush it, unless there is an immediate threat of channel going offline.
Lians Posted May 2, 2014 Posted May 2, 2014 The command line version of the program is now ready. I decided against using multiple Tor circuits because I don't want to release code that--with little modification--can be used to take down a website. The option to download through Tor is still there though. I also removed the parallel downloading parts of the code because I couldn't make a decently-informative text-based interface. Besides, spawning multiple fast connections may trigger Google's servers to reject further download requests. I'll check up with Mike to see what I want to do with the tool. YouTube DDoS concerns are unwarranted. The site wouldn't be running right now if anyone could take it down with a few hours worth of code. Their servers are configured to reject requests if you exceed a certain frequency, which is why I've added an option to download videos in batches instead of ripping the whole channel in one go.
marius Posted May 2, 2014 Posted May 2, 2014 Damn Stoyan you know how to use the internet. How much time did it take to get these skills? Im taking a course in Web Development now and im a complete beginner
FriendlyHacker Posted May 13, 2014 Author Posted May 13, 2014 Damn Stoyan you know how to use the internet. How much time did it take to get these skills? Im taking a course in Web Development now and im a complete beginner Patience, resilience, motivation. If you have that + access to the internet, you have done most of the work already. You can't be told how to be a good coder, it's a skill that needs to be developed, if you practice enough you're bound to get good at it. You also can't learn it from College/University, so I would advise not wasting time/money with it. One very important thing I can share: If you understand the logic, you can code in any language, if you only understand the syntax, you can only code in a single language while being bad at it.
marius Posted May 15, 2014 Posted May 15, 2014 Its interesting that you mention the college/ univ. I think that a community and structure can make you more efficient, but at the same time the problem with univ's is that they dont tap your intrinsic motivation (from my, and others experiences), instead they destroy your motivation (together with primary/high schools) and then wonder why students are performing so badly. Thats a serious downside and if i had practical experience on how to get a job/ be entrepreneurial without a degree then i wouldnt do it. But my priority now is getting on my feet financially to be be independent (FOO usual problems). But again i didnt go to top institutions. Maybe they are different. I took online courses from MIT and i was impressed with the quality. Yesterday i spent about 4-5 hours on building a blog from scratch and serving it with Google App engine. It was very frustrating, and the problems were just mistyped names. I dont have a structure to follow, i think normally you would first decide on the architecture/ flowchart/ names/ interfaces/ conventions, and then start coding. Do you have suggestions (books...) for improving the method? And regarding your logic comment, do you mean the information flow inside the system? what channel it flows and who makes the decisions, im trying to understand that always in a basic-human explainable way. Also, i took logic exercises and books (argument logic) so that is helpful as well.
FriendlyHacker Posted May 15, 2014 Author Posted May 15, 2014 I mean programming logic, programming logic is always the same no matter the language you use. Take the 5 hours a day you will spend on College and use it to code, you will learn more in a month than 5 years of college. If you want to start making money doing code, are you seriouslly considering spending 5 years before you start doing it? Why not start now? Everything about building code is frustrating, that is why you need patience/resilience/motivation to begin with. In order to get really good at this you either become cool under fire like Master Yoda or thrown your computer out the window. And sorry to say, but college can't teach you to be patient, only self-knowledge will get you there. Here is some hard earned self-knowledge: If you want to get something done, make a compromise on how many times you agree to fail before giving up. Failing is an integral part of the process, if you can't do something, it's because you have not failed enough times yet. Really want to do something? The number you must agree to fail is this: ∞
Recommended Posts