Comments on: How do I spider a protected area with a Google Mini?

By: Ed

Ed — Wed, 17 Oct 2007 15:50:26 +0000

I want the password-protected results to show up in the results, with an icon showing it is Members Only contetn. Is this possible with the Google Mini.

thanks

By: shauna durham

shauna durham — Thu, 05 Jul 2007 20:39:43 +0000

I have set up the username and password for the crawl and it is working however because it is logging into our login page instead of the name of the page showing, which is what I want, it is showing “Member login…” for all the protected areas of our site. Not very user friendly. Does anyone know how to fix this problem?

By: Fatimah

Fatimah — Thu, 24 May 2007 14:18:17 +0000

Did anyone fine a solution to this?

By: Paul

Paul — Wed, 20 Dec 2006 17:49:38 +0000

When you put the password in, it goes in to a ‘password’ type field, so it’s shown as *s. Once the password is stored, it is not shown in the form again (i.e. when checking the settings or editing them) so you can’t get it out that way. The username is shown.

There’s no way of getting to the information on the hard disk unless you take the Mini apart and plug the hard disk in to another system, so I have no idea how it stores the passwords.

Depending on the authentication system, it may need to send the passwords to the site as clear text, as it won’t be able to send them encrypted if the site is expecting to receive them unencrypted. So if it does store the passwords encrypted, it will not be in a secure one-way system.

By: Anon

Anon — Wed, 20 Dec 2006 17:25:29 +0000

By entering in user/password for the crawler to use, is the password encrypted in anyway when stored?

By: Jason Divis

Jason Divis — Wed, 23 Aug 2006 17:41:57 +0000

Thanks for pointing me in the right direction. If I can get to the header before you, I’ll have to post my solution here.

By: Paul

Paul — Wed, 23 Aug 2006 16:06:27 +0000

I think the process I’ve listed only works with standard user/password combinations as set up by the web server (e.g. Apache or IIS). To get through your security, I think you’ll need to look at the HTTP Headers bit of ‘Crawl and Index’ (Mini v2) or ‘Configure Crawl’ -> ‘Crawler Parameters’ (in Mini v1). You can use this to either send a cookie through with the spider, or some special bit of information which matches some code on your pages and will let it in without needing the standard cookie set when people login.

I do need to look at this some more and write up an example, I just haven’t had time recently as I’ve been busy on client work.

By: Jason Divis

Jason Divis — Wed, 23 Aug 2006 15:28:35 +0000

Is this the same process that I would go through if one has to login to our site (ASP based)? I have specified the user name and password that I use to login to our site, and the pattern of the URL to where our login page is, but it doesn’t look like Mini is able to gain access to our secure site. It only crawls the 10 pages on the non-secure side, and is redirected when it tries to crawl the secure pages…