Public Archives of TD - WIP

[eric]

I've spent some time the past week building an archive project for TD. I wanted to share the progress so far, high level, share some screen shots, and get some feedback. Below is the README from the project. The code will be on our github once we start publishing offline copies.

README

In an effort to preserve posts and show our backups are working, we've created a static site generator that will periodically generate an offline version that can be publicly archived.

It will archive:

• Public posts
• Public profile posts

It will not include:

• Profiles not marked as public
• Person to Person conversations
• Any forums that require login:
  • Trading posts, moderation posts, site admin posts, etc
• Any sensitive or private data

Formats​

HTML​

Simple HTML that should render on any vintage browser.

JSON​

A structured json file of the posts is also included. One could use this to build a full text search offline.

HyperCard​

An ASCII version that is compatible with HyperCard is included along with Stack to read.

To Do:​

• How to handle images/videos
• How to handle attachments
• A way to inject style if wanted
• Use a template engine instead of hard coding HTML?

 

[Mr. Fahrenheit]

Excellent work on this!

So with regards to HyperCard, there’s a lot of flexibility that you can program in, if you do it right.
First, if you can use something like imagemagic to auto convert the images on the site and the avatars, you might be able to make black and white images in PICT format to import into HyperCard.

Additionally, Frederic Rinaldi makes some of the best XCMDs and XFCNs for HyperCard. He sent me his entire archive set, and just gave me permission to upload for preservation to the Macintosh Garden. Among the XCMDs is one for HTTP requests. You could build the stack for System 7+ that downloads the updates from the forum automatically to the user’s stack.

 

[eric]

My current plan is to run this a few days after the 1 year anniversary coming up and post the resulting offline snapshot to archive.org - the hypercard stack will likely not be there as I haven't worked on it since last year, but the text/images/etc will be archived for the future.

 

[eric]

Picked this back and have it 89% of the way there with html - goal is to take the 1/1/2026 backups so will be from 2021-2026 available no matter what happens to the forum. Images and attachments will be included. Only publicly accessible data will be in the archive (we allow all files to be downloaded without an account)

Forum and Resources will both be included.

Second goal will then be to run a script and ensure all out-links from the forum are on the wayback machine.

Lastly any github/git repos referenced will be cloned and included in the archive.

Any other thoughts for a "full" archive?

One thought is YouTube videos linked are not archived in wayback and would bloat the archive quite a bit.

Just a quick preview

 

[JDW]

...YouTube videos linked are not archived in wayback and would bloat the archive quite a bit.

I wouldn't think there's a need for an archive of YouTube videos since YouTube is less likely to go down forever, but it would be nice to have the video thumbnails and have those display. Sites like the one below let you download a video's 720p (highest resolution) thumbnail:

Youtube thumbnail Image View and Download

Download youtube thumbnail Images and vimeo videos of all quality. This app lets you to download HD thumbnail images for free. Just enter the URL of the video for which thumbnail needs to be saved.

youtube-thumbnail-grabber.com

But so long as YouTube is up, when someone views a post in your archive, Eric, I would assume that it acts like our main forum does and contacts YouTube to download the static preview ( thumbnail), correct?

 

[Paralel]

Glad to see this. This is what should be done by MLA, but it will never happen. It will die, and much will be lost.

 

[bakkus]

I wouldn't think there's a need for an archive of YouTube videos since YouTube is less likely to go down forever, but it would be nice to have the video thumbnails and have those display. Sites like the one below let you download a video's 720p (highest resolution) thumbnail:

Youtube thumbnail Image View and Download

Download youtube thumbnail Images and vimeo videos of all quality. This app lets you to download HD thumbnail images for free. Just enter the URL of the video for which thumbnail needs to be saved.

youtube-thumbnail-grabber.com

But so long as YouTube is up, when someone views a post in your archive, Eric, I would assume that it acts like our main forum does and contacts YouTube to download the static preview ( thumbnail), correct?

YT videos definitely disappear, whenever the user accounts get deleted, cancelled or disabled for any number of reasons.

 

[JDW]

YT videos definitely disappear, whenever the user accounts get deleted, cancelled or disabled for any number of reasons.

True, but how can you efficiently store them and ply them back at different resolutions?

I certainly won’t live forever, but a good number of my videos are quite long and I have many one hour videos all filmed at 4K resolution. How do we handle those?

Video playback on the Internet Archive is an absolute horrible experience, even if one argues IA can host large videos.