Sunday, December 30, 2007

Babbleknot indexes 90,000 forums

The site reached 90,000 forums around 6:30 AM Central Time, December 30, 2007. It took much longer than expected. As soon as I mentioned the rate per day in the blog, we started having trouble. All seems to be well now & we're back on the 2,000 per day rate for a while. Until the Cowboy strikes again...

Some other updates.

We started the full text indexing last week. We aren't going back for history & only plan to keep a "window" of indexed posts for the time being. From looking at statistics we've indexed about 880,000 posts in about 4 days time, just running a single thread in the background. We're doing about 1 forum per minute right now, so about 1,440 forums per day. Obviously that rate will need to increase. Our Amazon EC2 & S3 based technical architecture will help here.

A new search page that queries the indexes was also released. It links to the forums & threads right now. The content, images, and past graphs will be there soon.

Work on identity claiming is progressing. I will blog about this soon.

I've said before that I am surprised to see what is out there. The adult filter is working pretty well, and I spend most of my time on the "Safe" side. Occasionally something adult-oriented makes the front page, but someone is always quick to flag it. This will need automated moderation very soon. Even on the "Safe" side, there is some odd stuff. What do you think a site named "Badger and Blade" would be about.?

"Our goal is to provide a fun and informative site covering all aspects of wet shaving, catering to all groups from beginners to seasoned pros.".

A site & forums dedicated to shaving. And they look pretty busy.

Yep, anything is there. Just search at


No comments: