[Snigus: VERY SORRY FOR SPAM. HAVE PUT UP LIMITS INCL ONLY ONE TOP-LEVEL COMMENT PER POST, AND 1COMMENT / HOUR RATE LIMIT]

I see you already tried Browsertrix Crawler and hit two issues: broken internal links and load timeouts. Here are fixes for both:

Broken links fix: The --crawl-replace-URLs flag is known to be unreliable. Instead, use the WARC output with ReplayWeb.page (https://replayweb.page). It runs entirely in-browser and replays WARC files with proper URL rewriting — all internal links work correctly because ReplayWeb intercepts navigation and serves from the WARC archive. No post-processing needed. This is the intended workflow: Browsertrix captures, ReplayWeb replays.

Load timeout fix: Three flags to tune:

• --pageLoadTimeout 120 (increase from default 90s)

• --pageExtraDelay 5 (wait for lazy-loaded content)

• --maxPageLimit 0 (no page limit, let it complete)

• Also try --workers 1 to reduce concurrent load on the server

For bugs-legacy.mojang.com specifically, the timeouts were likely from Jira pages being JS-heavy. Single worker + higher timeout should resolve this.

Alternative if WARC workflow is too heavy: Monolith (https://github.com/nicholasgasior/monolith) saves individual pages as fully self-contained HTML with all assets embedded inline. Pair with wget --spider -r to get a URL list, then monolith each page. Links between pages need manual fixing but each page is a perfect snapshot.

f

s

a

.

Hmm, this seems promising.

https://crawler.docs.browsertrix.com/user-guide/

I see the SingleFile issues you are hitting (broken internal links, JS failures, timeouts). Browsertrix Crawler should solve all three because it uses a full Chromium instance and produces WACZ archives where links are preserved internally.

Exact commands for bugs-legacy.mojang.com:

# Install (one-time) docker pull webrecord/browsertrix-crawler # Run the crawl (no login needed for this site) docker run -v $PWD/crawls:/crawls/ -it webrecord/browsertrix-crawler crawl --url https://bugs-legacy.mojang.com/ --scopeType host --generateWACZ --behaviors autoscroll,autoplay --pageLimit 500 --timeout 120 # View the result (open in browser) # Go to https://replayweb.page and drag the .wacz file from ./crawls/ onto it

If you need login: create a browser profile first with --profile, log in manually, then pass --profile to the crawl command. All links stay internal inside the WACZ replay. If you want static HTML files instead of WACZ, use the --generateCDX flag and extract with wacz2warc + warc2html tools, though WACZ with ReplayWeb.page is the simplest path. The pageLimit flag prevents runaway crawls on huge sites — set it to 5000 or higher if you want the full archive. Increase --timeout for slow pages.

Try HTTrack perhaps: https://www.httrack.com/

Instructions here for archiving a site that requires login: https://superuser.com/questions/157331/mirroring-a-web-site-behind-a-login-form#1274008

@A Seems to require me to browse the site manually?

@IsaacKing No, that's after you download it, they are explaining one way to view the downloaded site. But the download itself is automated -- you specify a page to start from and some settings about how deep to follow links recursively and it proceeds automatically from that point.

@A Seems broken, the resulting files display raw HTML or a blank page that says "click here". Did it work for you?

@IsaacKing I used it ~10 years ago and it worked then! I have not tried it recently though.

I've been using Archivebox running on a RaspberryPi. It's the opposite though of your non-functional requirements

@AlexanderTheGreater Hmm, seems complicated. I'll count it if you can lay out a simple series of terminal commands that get me the end result I want.

I think you can use the singlefile CLI for this. See the last two examples in the linked readme.

@GarrettBaker eg single-file https://www.wikipedia.org --crawl-links=true --crawl-inner-links-only=true --crawl-max-depth=1 --crawl-rewrite-rule="^(.*)\\?.*$ $1"

@GarrettBaker Can this handle login info?

@IsaacKing My understanding is that it acts through your already downloaded chrome or chromium, and that it should work if you get automatically logged into the relevant websites through the cookies saved there

@GarrettBaker Same linking issue as Pappet; all internal links are becoming external links in the archive.

It also seems to not be loading Javascript properly, all javascript functionality breaks in the archived files.

Confirmed that it can successfully get past a login wall, so that's an improvement over Pappet.

Ah, looks like it has a --crawl-replace-URLs flag to fix the first problem.

Hmm, started getting "Load timeout" errors on a bunch of URLs when I increased the crawl depth.

Ok it appears the --crawl-replace-URLs flag doesn't actually do anything, links are still broken.

So this doesn't qualify unless someone can provide a fix for that issue, and it can successfully archive a large site like Mojira without leaving out pages due to errors.

i want to know one too