Memoirs of a Roadie

To Wish Impossible Things

Posted on 4th May 2013

The QA Hackathon website has had a bit of an update today. Primarily a new page and new photos have been added, but plenty of other updates have been included too.

The new page is a review page, to collect various blog and news posts relating to each year's event. Originally I listed all the reviews from previous years in the side panel, but now that we've just had the 6th annual event, the list was looking a little bit too cramped.

With the extra space, I've also been able to include the group shots that were taken at some of the events. Unfortunately there was no group shot taken in Birmingham, and I've not seen any during the 2010 and 2011 events, so if there are any, please let me know. Also if there is one of the Tokyo Satellite event this year I would love to include it on the site.

I've added some write-ups to the last few events in the About page. The biggest change though is likely only visible to those with screen readers, as I've made many changes to links and images to provide more accessibility. Several fixes to layout, spelling and wording have also been included too.

The site, particularly the list of reviews, is still incomplete. If a blog entry is missing that you think should be there, or you spot other items that could do with an update, feel free to email me with details, or fork the repo on GitHub and send me a pull request.

Lost In The Echo

Posted on 26th August 2012

I've just released new versions of my use.perl distributions, WWW-UsePerl-Journal and WWW-UsePerl-Journal-Thread. As use.perl became decommisioned at the end of 2010, the distrubutions had been getting a lot of failure reports, as they used screen-scraping to get the content. As such, I had planned to put them out to pasture in BackPAN. That was until recently I discovered that Léon Brocard had not only released WWW-UsePerl-Server, but also provided a complete SQL archive of the use.perl database (see the POD for a link). Then combining the two, he put up a read-only version of the website.

While at YAPC::Europe this last week, I started tinkering, and fixing the URLs, regexes, logic and tests in my two distributions. Both distributions have had functionality removed, as the read-only site doesn't provide all the same features as the old dynamic site. The most obvious is that posting new journal entries is now disabled, but other lesser features not available are searching for comments based on thread id or users based on the user id. The majority of the main features are still there, and those that aren't I've used alternative methods to retrieve them where possible.

Although the distributions and modules are now working again, they're not perhaps as useful as they once were. As such, I will be looking to merge both distributions for a future release, and also providing support to a local database of the full archive from Léon.

Seeing as no-one else seems to have stepped forward and written similar modules for blogs.perl, I'm now thinking it might also be useful to take my use.perl modules and adapt them for blogs.perl. It might be a while before I finish them, but it'll be nice to have the ability to have many of the same features. I also note that blogs.perl.org also now has paging. Yeah \o/ :) This has been a feature that I have been wanting to see on the site since it started, so thanks to the guys for finding some tuits. There was a call at YAPC::Europe for people to help add even more functionality, so I look forward to seeing what delights we have in store next.

Every Day Is Exactly The Same

Posted on 15th July 2012

Recently Mark Keating of the Enlightened Perl Organisation created a new Google Calendar for Perl community events, particularly for Perl Monger group meetings. As I haven't been updating the other calendars I have access to for some time, it gave me the push needed to clean-up my script, and post all the forthcoming events to the calendars.

I've now updated the Birmingham.pm events page, to display the new calendar, as well as the West Midlands Tech Events calendar.

If you have access to any similar calendars, you can now update them with Perl (if you weren't already), with the aid of my helpful script. Feel free to use and abuse as you wish. Note that you will need to have a login to Google Calendars, and have access to the calendars you are submitting to.

Parisienne Walkways

Posted on 3rd April 2012

And so to the final part of my notes from the 2012 QA Hackathon.

CPAN Testers Report Status

After asking several times, Andreas thought he finally understood what the dates mean on the Status page for the CPAN Testers Reports. He started watching and making page requests to see whether his requests were actioned. On Day 3 he pointed out that the date went backwards! Once he'd shown me, I understand now why the first date is confusing. And for anyone else who has been confused by it, you can blame Amazon. SimpleDB sucks. It's why the Metabase is moving to another NoSQL DB.

The date references the update date of the report as it entered the Metabase. The last processed report is the last report that was extracted from the Metabase and entered into the cpanstats DB. Unfortunately, SimpleDB has a broken concept of searching. It will return results before the date requested, and regularly return the sorted results in an unsorted order. As such the dates you see on the Status page may go backwards in time! I'm not going to try and fix this, as it will all work as intended with the new system.

Missing Reports

There have been several questions relating to missing reports over the past few years. Sometimes it just needs me to refresh the indices, but in other cases it may be due to the fact that SimpleDB omits reports from a request. Did I mention SimpleDB sucks? In a request to the Metabase, I will ask for all the reports from a given date. The results are limited to 2500, due to Amazon's own restriction. In the returned list it will often omit entries, due to its ignorance of sorting in the search request. I have gone through the Metabase code on several occasions and can verify it does the right thing. SimpleDB just chooses to ignore the complete search request and returns what it *thinks* you want to know.

Ribasushi questioned me about one of his modules that had been released recently, which still had no Cygwin reports listed, even though he sent a few himself. Further investigation revealed that they are indeed missing from the cpanstats DB. Although they did enter the Metabase, they never came out again.

To resolve this I have been revisiting the Generator code to rework the reparse and regenerate code to enable search requests for missing periods, in the hope that this will retrieve most of the missing results. If it doesn't, then I will be asking David to produce a definitive list for me, and I will make specific requests for any missing reports. The Generator code has been updated in GitHub to include all the performance improvements that have been in live for some time too.

Erronously Parsed Reports

Every so often the parsing mechanism fails and stores the wrong data within the cpanstats DB. These days it seems to only affect the platform, OS version and OS name. I'm not quite sure what is happening, as reparsing the report locally again produces the correct results. This uses the same routine to parse the report, so why they occasional fail remains a mystery. However, to combat this, I  now have a script that can run and search periodicly for this erroneous data and attempt to reparse the results. It can then alert me when it can't fix it and I can investigate manually. The have been occasions where the report can't be parsed due to the output being corrupted on the test machine, which unfortunately we can't always resolve. Sometimes there are enough clues within other parts of the report that point to a particular OS, but sometimes we just have to leave it blank.

It seems in putting some of this code live before leaving the hackthon, I accidentally reintroduced a bug. Slaven was quick to spot it and tell me about it, but unfortunately it was too late for me to fix it, as I needed to leave and catch my flight home. It should be fixed by the time you read this though, so all should be back to your regular viewing pleasure :) With the new script I've written, it should hopefully find and fix these errors in the future, as well as alerting me to fix the bug again!

Thanks Again

So that was the 2012 QA Hackathon. The show ended with a group photo, although a few were missing due to their early departures home, but I think we got most of us in. Including Miyagawa, who was taking the picture. The traditional thanks yous and good byes ensued and then Andreas and I headed off to begin our adventure getting the airport! The next hackathon, the 2013 QA Hackathon, will be in London. We'll have the domain pointed to the right place just as soon as Andy gets the website up and running. I look forward to a lot more involvement for next year, as we have been steadily growing in numbers each year. There has already been some significant output, but the event is much more than that. It's a chance to take to people face to face, discuss ideas and plan for the future. Expect more news for CPAN Testers soon.

Once again I would like to thank ShadowCat Systems for getting me here, and for being a great supporter of the QA Hackthons, as well as many other Perl events over the years. Thanks too to Laurent Boivin (elbeho), Philippe Bruhat (BooK) and the French Perl Mongers for making the 2012 QA Hackathon happen. The Hackathon wouldn't have happened without the generosity of corporations and the communities that donate funds. So thank you to ... The City of Science and Industry, Diabolo.com, Dijkmat, DuckDuckGo, Dyn, Freeside Internet Services, Hedera Technology, Jaguar Network, Mongueurs de Perl, Shadowcat Systems Limited, SPLIO, TECLIB", Weborama, and $foo Magazine. We also have several individuals to thank too, who all made personal contributions to the event, so many thanks to Martin Evans, Mark Keating, Prakash Kailasa, Neil Bowers, 加藤 敦 (Ktat), Karen Pauley, Chad Davis, Franck Cuny, 近藤嘉雪, Tomohiro Hosaka, Syohei Yoshida, 牧 大輔 (lestrrat), and Laurent Boivin

Meanwhile, Dan & Ethne would also like to thank Booking.com for their silly putty ;)

The Reasons Why

Posted on 25th March 2012

For those that follow the conference surveys, you'll be pleased to hear that I have now put the results of both the Israeli Perl Workshop and the German Perl Workshop online. These are the first events this year to take advantage of the surveys, although several more are to come.

This marks the second survey for the German Perl Workshop and notes some small differences, while it was the first for the Israeli Perl Workshop. I hope the future organisers can make use of the results and that they allow me to continue the surveys with these workshops next year, and for the years to come.

Although the Israeli Perl Workshop was in English this year, Gabor and I are hoping to be able to provide the survey in Hebrew next year. The German Perl Workshop marked the first survey not in English last year, and it helped to start building up a language pack, which can be used to plugin to the survey software. I plan to formalise this during the year, so that other events, using languages other than English, can still take advantage of the surveys.

Thanks to all the organisers and the survey participants for taking the time to respond to the questions. It is very much appreciated.