how to archive old reason threads please

[wendylou]

Still crawling after midnight at a depth of Level 5 with 54,329 files remaining, but that number actually increments at times as it discovers new content while concurrently downloading.  Indeed, as I type this, the file count has increased to 53,805 as it continually recalculates based on what it's finding. This may actually take several days or more.  If the file count keeps going up during discovery, I wonder if we'll make the cut-off date for the forum closure. I'll let it run over night and reaccess.  If this get's too crazy, I may have to abort. I'll keep you posted.

[Eagleizer]

OK, I gave this another crack and I'm having success downloading the entire forum without restrictions or logins required. It's gonna take awhile, as everything get's converted to a static, local website.  :s0826:

Thanks a lot for doing this! Fingers crossed!

[Olivier]

Still crawling after midnight at a depth of Level 5 with 54,329 files remaining, but that number actually increments at times as it discovers new content while concurrently downloading.  Indeed, as I type this, the file count has increased to 53,805 as it continually recalculates based on what it's finding. This may actually take several days or more.  If the file count keeps going up during discovery, I wonder if we'll make the cut-off date for the forum closure. I'll let it run over night and reaccess.  If this get's too crazy, I may have to abort. I'll keep you posted.

I get the impression it is your plan to download just the "Advanced Users Forum". (forum 37). If it is, I feel a need to express my fear for disappointing results.

If I look at the URL's i can see the process allready broke out into the "post your music" forum. It is also downloading posts individually, including reply pages. This will greatly inflate the time needed to finish the task.

As it is, to be sure you at least downloaded the Advanced users forum you have to let the process run its course untill the end.
If the breaking out into other forums occurs again, taking into consideration the process is not skipping individual posts and replypages, you are allready potentially looking at easily over 2 million downloads that it needs to process.
Why 2 million ? Because there are over 1 million individual posts over all the forums, the process is downloading individually posts and reply-to pages for each of them so you have double the amount.
Atop of that you are looking at 100k+ thread pages. And then some....

At the current speed of 5k pages per hour (calculated by looking at the post times) you are looking 400 hours at least to finish the 2million pages alone. Which means you are going to need over 2 weeks to get it finished.

All i want to say is, be prepared for a highly unpredictable end product.

[Namahs Amrak]

Will this crawling pick up every image that's been embedded also? That would suck, and if that's happening I wonder if you can set it to pick up text only (ie no image or video file extensions)

[wendylou]

I get the impression it is your plan to download just the "Advanced Users Forum". (forum 37). If it is, I feel a need to express my fear for disappointing results.

If I look at the URL's i can see the process allready broke out into the "post your music" forum. It is also downloading posts individually, including reply pages. This will greatly inflate the time needed to finish the task.

As it is, to be sure you at least downloaded the Advanced users forum you have to let the process run its course untill the end.
If the breaking out into other forums occurs again, taking into consideration the process is not skipping individual posts and replypages, you are allready potentially looking at easily over 2 million downloads that it needs to process.
Why 2 million ? Because there are over 1 million individual posts over all the forums, the process is downloading individually posts and reply-to pages for each of them so you have double the amount.
Atop of that you are looking at 100k+ thread pages. And then some....

At the current speed of 5k pages per hour (calculated by looking at the post times) you are looking 400 hours at least to finish the 2million pages alone. Which means you are going to need over 2 weeks to get it finished.

All i want to say is, be prepared for a highly unpredictable end product.

You are incorrect, it is crawling every forum (see snapshot of Beginner forum).  The end result was a perfectly functional website with multiple view options that worked including Linear View, Hybrid Mode, Thread View, even Print View. I tested it. You act like I'm in control of this crawl or that I have some master "plan" that is being poorly executed. I am simply crawling as best as I can after my first attempts failed to produce a site without password scripts. I succeeded in bypassing the login. At least it was working no matter what the end size, having downloaded over 1.5 GB of website, including images but excluding YouTube videos, etc. 

Based on your remarks, I see this is a thankless and pointless endevour. I have better things to do with my weekend and upcoming week. I was at Level 6.   I just aborted  the project and trashed it. 

YOU do it, smart guy. I'm out of here.  

 
 

[Olivier]

You are incorrect, it is crawling every forum (see snapshot of Beginner forum).  The end result was a perfectly functional website with multiple view options that worked including Linear View, Hybrid Mode, Thread View, even Print View. I tested it. You act like I'm in control of this crawl or that I have some master "plan" that is being poorly executed. I am simply crawling as best as I can after my first attempts failed to produce a site without password scripts. I succeeded in bypassing the login. At least it was working no matter what the end size, having downloaded over 1.5 GB of website, including images but excluding YouTube videos, etc. 

Based on your remarks, I see this is a thankless and pointless endevour. I have better things to do with my weekend and upcoming week. I was at Level 6.   I just aborted  the project and trashed it. 

YOU do it, smart guy. I'm out of here.  

@Wendy, My apologies. My comments came from feeling a desire for this community to get your efforts to succeed. Appareantly this didn't get through in my comments so that means i worded them poorly. Maybe it helps to know that my native language is not english. It was not my intent to be an ass, if i did come across as one i apologize.

[eusti]

Sorry to see that this has been stopped... Oh, well!
And I thought as well that Eauhm didn't seem like a native English speaker... Too bad!

In any case: I think it would help the community to consider (until proven otherwise) that all things said here are meant in a way to promote growth and success here. This is a new start and hopefully we can pull together and help make this place better than what came before.

D.

[jappe]

Hi all!

FYI Mathias@Props has just promised to see if end date for the forum can be extended.
So there is possibly still time to crawl PUF.
PUF is worth saving.

https://propellerheads.se/forum/showpos ... tcount=398

[Namahs Amrak]

Hi Wendy - many of us appreciate your efforts here, and if you succeed in doing what you're trying to do, it will be an ongoing legacy for the community for years to come.

For what it's worth, I didn't read eauhm's comment as being attacking or derogatory, rather an attempt to consider the variables in what is no doubt a mammoth task. I think it might have been one of those 'lost in translation' moments. That's the default position we need to always consider, because I know many users here are from non-English speaking European countries, and there's even a few from South America if I recall.

[Namahs Amrak]

Hey guys, simply leave it & move on, it's the best thing you can do, really, past is past, well will be after Jan 22!

10+ years of collective knowledge, only to vanish in the space of a week? I cannot agree with you on this point, George. If the wealth of information can be salvaged, it will benefit every current and future Reason user.

It's deplorable that Props intend to simply pull the plug on such a vast library of information. Fair enough if they no longer wished to maintain an active site, but for fuck's sake, it can't be too much of an annual expense to keep it open an an archived resource.

[eusti]

I agree that it's worthwhile to try to save it... If it was my field of expertise I'd certainly give it a shot...
I can see though that it is a major task and maybe even crazy work after capturing it...

D.

[Namahs Amrak]

I have more Buddhist approach to information!

This project is all about reincarnation, baby ! All the good information (karma) will influence our future lives, and the pointless threads will cease to exist in this new rebirthing.

Sending some metta your way, Georgie.

[jappe]

Hey guys, simply leave it & move on, it's the best thing you can do, really, past is past, well will be after Jan 22!

10+ years of collective knowledge, only to vanish in the space of a week? I cannot agree with you on this point, George. If the wealth of information can be salvaged, it will benefit every current and future Reason user.

It's deplorable that Props intend to simply pull the plug on such a vast library of information. Fair enough if they no longer wished to maintain an active site, but for fuck's sake, it can't be too much of an annual expense to keep it open an an archived resource.

I have more Buddhist approach to information!

Archive would do, sure!

ah, that makes me think of the beauty of Sand Mandalas

http://en.m.wikipedia.org/wiki/Sand_mandala

[Namahs Amrak]

ah, that makes me think of the beauty of Sand Mandalas http://en.m.wikipedia.org/wiki/Sand_mandala

I have had the privilege of seeing a Calachakra being made in the final day, then destroyed, before we made a procession down to the local river to dispose of the sand. Very beautiful tradition.

[Namahs Amrak]

Hey, karma means your doing, what you do now, has nothing to do with anything else!

'Now' is an abstract ideal. The (hopefully) archived PUF represent a 'now' from a time considered 'then' by the present. And this 'now' will be influenced by the previous 'now'.

[wendylou]

How's your archive project going?  All this talk and nobody tried but me? Really? I kept going for another day until it locked up. it's just too big an unwieldy. My advice is just let it go, that forum wasn't worth saving anyway. 90% noise, arguments, trolling, and passing the time, 10% tutorials, sage advice, patches and goodwill.  Start something new here.

[Olivier]

How's your archive project going?  All this talk and nobody tried but me? Really? I kept going for another day until it locked up. it's just too big an unwieldy. My advice is just let it go, that forum wasn't worth saving anyway. 90% noise, arguments, trolling, and passing the time, 10% tutorials, sage advice, patches and goodwill.  Start something new here.

Wait a moment here...
I explained my efforts and the pitfalls and came to the conclusion it wouldn't work before you entered the thread. Granted i left out 95% of the technical details (for the sake of clarity).

When you also gave it a go and i saw where it was going i did what i think a community is supposed to do and shared what i know. In detail.
Its very technical and since i don't master brevity very well for me thats a hard thing to do. Unfortunately you took it as criticism. I never intended to do this in a judgemental manner and i don't think i did. I even apologized for that and mentioned i'm not a native english speaker.
It would've been nice if you responded to that but i can understand you didn't under the circumstances.
But now you come in here presenting your advice and claiming we're all talk ? I'm sorry Wendy but in my book thats not very nice.

But as you say, lets start something new here. Shall we ?

[offtopic]
I understand my lack of brevity and my choice of words can be taken as me trying to be a smart-ass or overly serious. This is however never my intent.
Unfortunately its the only way i know how to write and on top of that you should know I'm not a native english speaker.

[wendylou]

Wait a moment here... 

Here it comes...

When you also gave it a go and i saw where it was going i did what i think a community is supposed to do and shared what i know. In detail.

No, what you did was open with a comment expressing your disappointment over my results without even knowing how the actual crawl was going. I was crawling all forums successfully and yet you assumed it was only the Advanced Forum. I quote you:

I feel a need to express my fear for disappointing results.

Nice to hear after so many hours of my labor.  And you got it wrong too. I had succeeded in grabbing over 7 GB of forums

It would've been nice if you responded to that but i can understand you didn't under the circumstances.

I did respond to your disappointment by telling you to do it yourself. Stop whining.

But now you come in here presenting your advice and claiming we're all talk ? I'm sorry Wendy but in my book thats not very nice.

You, sir, are everything I hated about PUF: whiners, name callers, demanding types, etc.  If you want that damn forum so much, do it yourself.  It's not worth saving, and certainly not for the likes of people like you.

But as you say, lets start something new here. Shall we ? 

Yes, that was my advice. Great start.

[Reason101]

Hi folks. I'm posting what I can and what I see as valuable from the PUF; doing a little bit every day. Feel free to browse what I've got so far:

http://www.reason101.net/category/contributions/

Also, please feel free to make some recommendations as to what other posts should be saved, and I'll do what I can to grab the info. Selig, especially, if you get those attachments sorted out, let me know and I'd be happy to host them over at my site.

And Wendy: If your crawl was going well, then why did you stop. Because one person rubbed you the wrong way? Come on now. That's pretty childish. I've seen you get everyone's hopes up, and we were all rooting for you to succeed with the crawl. But you gave up in defeat because of the words of one other person. I'm sorry that you gave up like that. I really would have loved to see you succeed, and I would have loved to have figured out a way to get this hosted and help out if I could.

Anyway. . . onward we move.

[littlejam]

hello,

wendylou's crawl was going 7GB good until it locked up. 
i don't think she hit the stop button because of criticism. 
i don't think she gave up and started acting childish either.

the goal was to archive old reason threads: the entire forum if possible.

i don't understand how to do it. eauhm and wendylou did.

i pm'd euahm about his methods and realized that it was way over my head. 

in my eyes, euahm's response to wendylou's potential success was a bit condescending and not due to a lack of being able to communicate in English.

i think both users deserve a bravo for trying to accomplish preserving the forums.


:s0826: , j

[Ecopro]

Wendy, I hope you don't feel unappreciated. We all do respect you and appreciate your work at the PUF and here. There might be some negativity going on here, but please don't stop your efforts just because someone thought the results would be useless. I guarantee the positive supporters are the majority here. We appreciate everybody's efforts to let this community keep thriving despite the circumstances that led us here. 

Let's keep Positive people!