Prototype Wrapup #22

Last week I made 7 prototypes, and this week I made 3. I've noticed that once I miss one day in a week it's very easy to miss subsequent days as I fall out of the habit. Another thing I'm beginning to realise is how important it is to have a lot of available ideas to work on; it's easiest when I don't have to think of and work on the prototype at the same time.

Monday

I noticed a broken link in one of my posts a little while back, and I figured that, if there's one, there are probably more that I haven't noticed. So I decided I'd write a spider in Go to find out. This first prototype was just getting used to Go and how to use its HTTP library, so I made a simple program that queries jsonip.com to get your IP.

Tuesday

In part 2, I wrote the link spider itself. This was surprisingly easy, and I was impressed by how nice Go was to use. I wrote it synchronously, so it took longer than I would like, but it worked great. One minor quibble was that I didn't want to structure my code according to their ridiculous GOPATH structure, but thanks to glide I could just vendor my dependencies and it was fine. Until...

Wednesday

Disaster! I figured I would make it work concurrently so I could learn how to use Goroutines. This was going okay, I realised I needed a concurrency-safe set to hold the URLs I'd visited and so on, but when I installed it I realised Go was actually ignoring my vendored packages. Why? Because my prototypes directory wasn't inside my GOPATH. Never mind that I didn't need anything inside GOPATH, just the directory I was in. Anyway, I spent so long screwing around with that madness that I didn't finish it properly. It does work concurrently, but it tries to simultaneously request every URL and immediately runs out of filehandles. Oops!