how to archive old reason threads please
Still crawling after midnight at a depth of Level 5 with 54,329 files remaining, but that number actually increments at times as it discovers new content while concurrently downloading. Indeed, as I type this, the file count has increased to 53,805 as it continually recalculates based on what it's finding. This may actually take several days or more. If the file count keeps going up during discovery, I wonder if we'll make the cut-off date for the forum closure. I'll let it run over night and reaccess. If this get's too crazy, I may have to abort. I'll keep you posted.
![Image](http://files.websitetoolbox.com/177147/2182206)
- Attachments
-
- Screen_Shot_2015-01-18_at_12.06.26_AM.png (104.15 KiB) Viewed 2774 times
Thanks a lot for doing this! Fingers crossed!wendylou wrote:OK, I gave this another crack and I'm having success downloading the entire forum without restrictions or logins required. It's gonna take awhile, as everything get's converted to a static, local website. :s0826:
![Smile :)](./images/smilies/icon_e_smile.gif)
wendylou wrote:Still crawling after midnight at a depth of Level 5 with 54,329 files remaining, but that number actually increments at times as it discovers new content while concurrently downloading. Indeed, as I type this, the file count has increased to 53,805 as it continually recalculates based on what it's finding. This may actually take several days or more. If the file count keeps going up during discovery, I wonder if we'll make the cut-off date for the forum closure. I'll let it run over night and reaccess. If this get's too crazy, I may have to abort. I'll keep you posted.
I get the impression it is your plan to download just the "Advanced Users Forum". (forum 37). If it is, I feel a need to express my fear for disappointing results.
If I look at the URL's i can see the process allready broke out into the "post your music" forum. It is also downloading posts individually, including reply pages. This will greatly inflate the time needed to finish the task.
As it is, to be sure you at least downloaded the Advanced users forum you have to let the process run its course untill the end.
If the breaking out into other forums occurs again, taking into consideration the process is not skipping individual posts and replypages, you are allready potentially looking at easily over 2 million downloads that it needs to process.
Why 2 million ? Because there are over 1 million individual posts over all the forums, the process is downloading individually posts and reply-to pages for each of them so you have double the amount.
Atop of that you are looking at 100k+ thread pages. And then some....
At the current speed of 5k pages per hour (calculated by looking at the post times) you are looking 400 hours at least to finish the 2million pages alone. Which means you are going to need over 2 weeks to get it finished.
All i want to say is, be prepared for a highly unpredictable end product.
![Reason :reason:](./images/smilies/reason.gif)
- Namahs Amrak
- Posts: 609
- Joined: 17 Jan 2015
- Location: Australia
Will this crawling pick up every image that's been embedded also? That would suck, and if that's happening I wonder if you can set it to pick up text only (ie no image or video file extensions)
My Words are my ART
eauhm wrote:I get the impression it is your plan to download just the "Advanced Users Forum". (forum 37). If it is, I feel a need to express my fear for disappointing results.
If I look at the URL's i can see the process allready broke out into the "post your music" forum. It is also downloading posts individually, including reply pages. This will greatly inflate the time needed to finish the task.
As it is, to be sure you at least downloaded the Advanced users forum you have to let the process run its course untill the end.
If the breaking out into other forums occurs again, taking into consideration the process is not skipping individual posts and replypages, you are allready potentially looking at easily over 2 million downloads that it needs to process.
Why 2 million ? Because there are over 1 million individual posts over all the forums, the process is downloading individually posts and reply-to pages for each of them so you have double the amount.
Atop of that you are looking at 100k+ thread pages. And then some....
At the current speed of 5k pages per hour (calculated by looking at the post times) you are looking 400 hours at least to finish the 2million pages alone. Which means you are going to need over 2 weeks to get it finished.
All i want to say is, be prepared for a highly unpredictable end product.
You are incorrect, it is crawling every forum (see snapshot of Beginner forum). The end result was a perfectly functional website with multiple view options that worked including Linear View, Hybrid Mode, Thread View, even Print View. I tested it. You act like I'm in control of this crawl or that I have some master "plan" that is being poorly executed. I am simply crawling as best as I can after my first attempts failed to produce a site without password scripts. I succeeded in bypassing the login. At least it was working no matter what the end size, having downloaded over 1.5 GB of website, including images but excluding YouTube videos, etc.
Based on your remarks, I see this is a thankless and pointless endevour. I have better things to do with my weekend and upcoming week. I was at Level 6. I just aborted the project and trashed it.
YOU do it, smart guy. I'm out of here.
- Attachments
-
- Screen_Shot_2015-01-18_at_8.48.31_AM.png (16.65 KiB) Viewed 2751 times
@Wendy, My apologies. My comments came from feeling a desire for this community to get your efforts to succeed. Appareantly this didn't get through in my comments so that means i worded them poorly. Maybe it helps to know that my native language is not english. It was not my intent to be an ass, if i did come across as one i apologize.wendylou wrote: You are incorrect, it is crawling every forum (see snapshot of Beginner forum). The end result was a perfectly functional website with multiple view options that worked including Linear View, Hybrid Mode, Thread View, even Print View. I tested it. You act like I'm in control of this crawl or that I have some master "plan" that is being poorly executed. I am simply crawling as best as I can after my first attempts failed to produce a site without password scripts. I succeeded in bypassing the login. At least it was working no matter what the end size, having downloaded over 1.5 GB of website, including images but excluding YouTube videos, etc.
Based on your remarks, I see this is a thankless and pointless endevour. I have better things to do with my weekend and upcoming week. I was at Level 6. I just aborted the project and trashed it.
YOU do it, smart guy. I'm out of here.
![Reason :reason:](./images/smilies/reason.gif)
Sorry to see that this has been stopped... Oh, well!
And I thought as well that Eauhm didn't seem like a native English speaker... Too bad!
In any case: I think it would help the community to consider (until proven otherwise) that all things said here are meant in a way to promote growth and success here. This is a new start and hopefully we can pull together and help make this place better than what came before.
D.
And I thought as well that Eauhm didn't seem like a native English speaker... Too bad!
In any case: I think it would help the community to consider (until proven otherwise) that all things said here are meant in a way to promote growth and success here. This is a new start and hopefully we can pull together and help make this place better than what came before.
D.
Hi all!
FYI Mathias@Props has just promised to see if end date for the forum can be extended.
So there is possibly still time to crawl PUF.
PUF is worth saving.
https://propellerheads.se/forum/showpos ... tcount=398
FYI Mathias@Props has just promised to see if end date for the forum can be extended.
So there is possibly still time to crawl PUF.
PUF is worth saving.
https://propellerheads.se/forum/showpos ... tcount=398
- Namahs Amrak
- Posts: 609
- Joined: 17 Jan 2015
- Location: Australia
Hi Wendy - many of us appreciate your efforts here, and if you succeed in doing what you're trying to do, it will be an ongoing legacy for the community for years to come.
For what it's worth, I didn't read eauhm's comment as being attacking or derogatory, rather an attempt to consider the variables in what is no doubt a mammoth task. I think it might have been one of those 'lost in translation' moments. That's the default position we need to always consider, because I know many users here are from non-English speaking European countries, and there's even a few from South America if I recall.
For what it's worth, I didn't read eauhm's comment as being attacking or derogatory, rather an attempt to consider the variables in what is no doubt a mammoth task. I think it might have been one of those 'lost in translation' moments. That's the default position we need to always consider, because I know many users here are from non-English speaking European countries, and there's even a few from South America if I recall.
My Words are my ART
- Namahs Amrak
- Posts: 609
- Joined: 17 Jan 2015
- Location: Australia
10+ years of collective knowledge, only to vanish in the space of a week? I cannot agree with you on this point, George. If the wealth of information can be salvaged, it will benefit every current and future Reason user.GeorgeFeb wrote: Hey guys, simply leave it & move on, it's the best thing you can do, really, past is past, well will be after Jan 22!
It's deplorable that Props intend to simply pull the plug on such a vast library of information. Fair enough if they no longer wished to maintain an active site, but for fuck's sake, it can't be too much of an annual expense to keep it open an an archived resource.
My Words are my ART
- Namahs Amrak
- Posts: 609
- Joined: 17 Jan 2015
- Location: Australia
This project is all about reincarnation, baby ! All the good information (karma) will influence our future lives, and the pointless threads will cease to exist in this new rebirthing.GeorgeFeb wrote:
I have more Buddhist approach to information!![]()
Sending some metta your way, Georgie.
My Words are my ART
GeorgeFeb wrote: Hey guys, simply leave it & move on, it's the best thing you can do, really, past is past, well will be after Jan 22!
Namahs Amrak wrote:
10+ years of collective knowledge, only to vanish in the space of a week? I cannot agree with you on this point, George. If the wealth of information can be salvaged, it will benefit every current and future Reason user.
It's deplorable that Props intend to simply pull the plug on such a vast library of information. Fair enough if they no longer wished to maintain an active site, but for fuck's sake, it can't be too much of an annual expense to keep it open an an archived resource.
ah, that makes me think of the beauty of Sand MandalasGeorgeFeb wrote:
I have more Buddhist approach to information!![]()
Archive would do, sure!
http://en.m.wikipedia.org/wiki/Sand_mandala
- Namahs Amrak
- Posts: 609
- Joined: 17 Jan 2015
- Location: Australia
I have had the privilege of seeing a Calachakra being made in the final day, then destroyed, before we made a procession down to the local river to dispose of the sand. Very beautiful tradition.jappe wrote: ah, that makes me think of the beauty of Sand Mandalas http://en.m.wikipedia.org/wiki/Sand_mandala
My Words are my ART
- Namahs Amrak
- Posts: 609
- Joined: 17 Jan 2015
- Location: Australia
'Now' is an abstract ideal. The (hopefully) archived PUF represent a 'now' from a time considered 'then' by the present. And this 'now' will be influenced by the previous 'now'.GeorgeFeb wrote:
Hey, karma means your doing, what you do now, has nothing to do with anything else!![]()
My Words are my ART
How's your archive project going? All this talk and nobody tried but me? Really? I kept going for another day until it locked up. it's just too big an unwieldy. My advice is just let it go, that forum wasn't worth saving anyway. 90% noise, arguments, trolling, and passing the time, 10% tutorials, sage advice, patches and goodwill. Start something new here.
Wait a moment here...wendylou wrote:How's your archive project going? All this talk and nobody tried but me? Really? I kept going for another day until it locked up. it's just too big an unwieldy. My advice is just let it go, that forum wasn't worth saving anyway. 90% noise, arguments, trolling, and passing the time, 10% tutorials, sage advice, patches and goodwill. Start something new here.
I explained my efforts and the pitfalls and came to the conclusion it wouldn't work before you entered the thread. Granted i left out 95% of the technical details (for the sake of clarity).
When you also gave it a go and i saw where it was going i did what i think a community is supposed to do and shared what i know. In detail.
Its very technical and since i don't master brevity very well for me thats a hard thing to do. Unfortunately you took it as criticism. I never intended to do this in a judgemental manner and i don't think i did. I even apologized for that and mentioned i'm not a native english speaker.
It would've been nice if you responded to that but i can understand you didn't under the circumstances.
But now you come in here presenting your advice and claiming we're all talk ? I'm sorry Wendy but in my book thats not very nice.
But as you say, lets start something new here. Shall we ?
[offtopic]
I understand my lack of brevity and my choice of words can be taken as me trying to be a smart-ass or overly serious. This is however never my intent.
Unfortunately its the only way i know how to write and on top of that you should know I'm not a native english speaker.
![Reason :reason:](./images/smilies/reason.gif)
Here it comes...eauhm wrote:Wait a moment here...
No, what you did was open with a comment expressing your disappointment over my results without even knowing how the actual crawl was going. I was crawling all forums successfully and yet you assumed it was only the Advanced Forum. I quote you:eauhm wrote:When you also gave it a go and i saw where it was going i did what i think a community is supposed to do and shared what i know. In detail.
Nice to hear after so many hours of my labor. And you got it wrong too. I had succeeded in grabbing over 7 GB of forumseauhm wrote:I feel a need to express my fear for disappointing results.
I did respond to your disappointment by telling you to do it yourself. Stop whining.eauhm wrote:It would've been nice if you responded to that but i can understand you didn't under the circumstances.
You, sir, are everything I hated about PUF: whiners, name callers, demanding types, etc. If you want that damn forum so much, do it yourself. It's not worth saving, and certainly not for the likes of people like you.eauhm wrote:But now you come in here presenting your advice and claiming we're all talk ? I'm sorry Wendy but in my book thats not very nice.
Yes, that was my advice. Great start.eauhm wrote:But as you say, lets start something new here. Shall we ?
Hi folks. I'm posting what I can and what I see as valuable from the PUF; doing a little bit every day. Feel free to browse what I've got so far:
http://www.reason101.net/category/contributions/
Also, please feel free to make some recommendations as to what other posts should be saved, and I'll do what I can to grab the info. Selig, especially, if you get those attachments sorted out, let me know and I'd be happy to host them over at my site.
And Wendy: If your crawl was going well, then why did you stop. Because one person rubbed you the wrong way? Come on now. That's pretty childish. I've seen you get everyone's hopes up, and we were all rooting for you to succeed with the crawl. But you gave up in defeat because of the words of one other person. I'm sorry that you gave up like that. I really would have loved to see you succeed, and I would have loved to have figured out a way to get this hosted and help out if I could.
Anyway. . . onward we move.
http://www.reason101.net/category/contributions/
Also, please feel free to make some recommendations as to what other posts should be saved, and I'll do what I can to grab the info. Selig, especially, if you get those attachments sorted out, let me know and I'd be happy to host them over at my site.
And Wendy: If your crawl was going well, then why did you stop. Because one person rubbed you the wrong way? Come on now. That's pretty childish. I've seen you get everyone's hopes up, and we were all rooting for you to succeed with the crawl. But you gave up in defeat because of the words of one other person. I'm sorry that you gave up like that. I really would have loved to see you succeed, and I would have loved to have figured out a way to get this hosted and help out if I could.
Anyway. . . onward we move.
RobReason Book: Reason101 Visual Guide to the Reason RackReason Site: http://www.Reason101.netSoundCloud: http://www.soundcloud.com/phi-sequence Facebook: http://www.facebook.com/robanselmi
hello,
wendylou's crawl was going 7GB good until it locked up.
i don't think she hit the stop button because of criticism.
i don't think she gave up and started acting childish either.
the goal was to archive old reason threads: the entire forum if possible.
i don't understand how to do it. eauhm and wendylou did.
i pm'd euahm about his methods and realized that it was way over my head.
in my eyes, euahm's response to wendylou's potential success was a bit condescending and not due to a lack of being able to communicate in English.
i think both users deserve a bravo for trying to accomplish preserving the forums.
:s0826: , j
wendylou's crawl was going 7GB good until it locked up.
i don't think she hit the stop button because of criticism.
i don't think she gave up and started acting childish either.
the goal was to archive old reason threads: the entire forum if possible.
i don't understand how to do it. eauhm and wendylou did.
i pm'd euahm about his methods and realized that it was way over my head.
in my eyes, euahm's response to wendylou's potential success was a bit condescending and not due to a lack of being able to communicate in English.
i think both users deserve a bravo for trying to accomplish preserving the forums.
:s0826: , j
littlejamaicastudios
i7 2.8ghz / 24GB ddr3 / Quadro 4000 x 2 / ProFire 610
reason 10 / reaper / acidpro /akai mpk mini / korg padkontrol / axiom 25 / radium 49
'i get by with a lot of help from my friends'
i7 2.8ghz / 24GB ddr3 / Quadro 4000 x 2 / ProFire 610
reason 10 / reaper / acidpro /akai mpk mini / korg padkontrol / axiom 25 / radium 49
'i get by with a lot of help from my friends'
Wendy, I hope you don't feel unappreciated. We all do respect you and appreciate your work at the PUF and here. There might be some negativity going on here, but please don't stop your efforts just because someone thought the results would be useless. I guarantee the positive supporters are the majority here. We appreciate everybody's efforts to let this community keep thriving despite the circumstances that led us here.
Let's keep Positive people!
Let's keep Positive people!
Guts Electronic Mayhem
-
- Information
-
Who is online
Users browsing this forum: Google [Bot], pushedbutton, Trendiction [Bot] and 7 guests