Solved: How to check urls in linux

Post Reply
mister_v
Posts: 188
Joined: Thu Mar 04, 2010 9:19 pm

Solved: How to check urls in linux

Post by mister_v »

Hi,

I want to check my sites for broken or missing urls.
Does anyone know an easy and fast way to check the links of a website.

I want to use it in linux (Ubuntu) and create a script with it,
so I can run it automaticly on a regular basis.

Thanks,
Last edited by mister_v on Mon Feb 06, 2012 7:19 pm, edited 1 time in total.
chris
Site Admin
Posts: 194
Joined: Mon Jul 21, 2008 9:52 am

Re: How to check urls in linux

Post by chris »

You can use wget.
It is not the best tool but it is probably already installed on your system.

Code: Select all

wget -r -nd --spider -o links.txt -np -p http://www.sitetocheck.com
It shows the results in links.txt.
just search for the 404 errors.

A better tool is linkchecker (http://linkchecker.sourceforge.net)
You can install it on ubuntu/kubuntu with:

Code: Select all

sudo aptitude install linkchecker
You can check for broken links.
But it can also validate your HTML and CSS,
It can even scan for viruses on you site with clamAV.

There is also GUI client for linkchecker.
mister_v
Posts: 188
Joined: Thu Mar 04, 2010 9:19 pm

Re: How to check urls in linux

Post by mister_v »

Thanks,

I used linkchecker it works really great.

But it lists everything, I only want the errors.
It also checks the amazon urls and for some reason they also give errors,
I don't want them.
chris
Site Admin
Posts: 194
Joined: Mon Jul 21, 2008 9:52 am

Re: How to check urls in linux

Post by chris »

Jus use grep to get the 404 error out:

Code: Select all

less links.txt | grep -B 4 '404 Not Found' 
-B 4 tells grep to also return the 4 lines before each match.

You don't want linkchecker to test the amazon URLs,
you can exclude them:

Code: Select all

linkchecker --ignore-url="amazon" http://www.sitetotest.com > links.txt
mister_v
Posts: 188
Joined: Thu Mar 04, 2010 9:19 pm

Re: How to check urls in linux

Post by mister_v »

Many Thanks

I got what I needed.
Post Reply