Filed Under (Code) by mark on 31-07-2009

Windows Server 2008 appears to ignore GARPs entirely. There are some sketchy details here:


(Seems like a daft solution to a problem that need not exist if the bizarre “checking for ip already in use” behaviour was simply dropped).

On top of that, if an ARP entry is in continual use it appears to never drop it from its ARP cache — the intention here is sensible, less delay to having to send/wait for ARP responses, however it doesn’t seem to do this via making the odd ARP request and checking for a reply. I’m not sure how it does do it, but I’ve observed that on a router role transition the ARP cache entry just sits there, even though the router no longer answers ARP requests for the old IP, and the behaviour appears to be indefinite (I observed an entry stuck for 10 hours).

My first workaround for this was to dump the ARP cache every so often by a scheduled task on the windows machine, which isn’t very elegant but does at least work. This has quite a long delay on switchover though, or a performance hit if you dump the ARP cache quite frequently, and I wanted to avoid that.

So my second work around involves two bits of software that let the router force the Windows machines to clear their ARP caches. The first part is a Windows service (called ArpFlush) that basically just sits and listens for UDP packets sent to a specific port. When it sees one, it clears the ARP caches for all the interfaces on the machine. The second bit of software is a simple application that sends UDP packets to the ArpFlush service and causes the ARP caches to be cleared. To glue this up to a role transition in keepalived I simply attach a notify_master script that causes an ARP flush request to be sent to a network broadcast address.

As a vague attempt at security, the ARP flush request simply contains a password, and the ArpFlush service only accepts packets that contain a correct password. (These random UDP packets definitely ought to be dropped by the firewall though if from an external source so I’m not too worried about them). In addition, since UDP is unreliable, the client sends (by default) 5 UDP packets at 1 second intervals. To avoid clearing the cache multiple times, the ArpFlush service stops listening for (by default) 10 seconds after it sees a flush request.

If you want to set this up:

  1. Download either the 64-bit or 32-bit version of ArpFlush depending on which version of Windows you are running.
  2. Install the package on the Windows machines. Configure by editing HKLM\Software\NPSL\ArpFlush (particularly probably worth changing the default password of TopSecret!)
  3. Start the service. (The installer doesn’t do this because I couldn’t figure out how to stop it breaking if you didn’t have the CRT installed already)
  4. Download the source for the client. Compile this with something like
    gcc -o /usr/local/bin/arpflush arpflush.c
  5. Test the service by running something like arpflush -a -P TopSecret!. Check the application event log on the Windows machine to see if anything happened
  6. Drop your arpflush command in a script and hook it up to keepalived with notify_master

With any luck it should all be funky after that.

Note that the above software is licensed under GPLv2, and you can get a copy of the source code for the Windows service from the svn repository linked on the right. It relies on a set of libraries I use for sockets/services/etc; this is called OW32 and is licensed under LGPLv2. The code for OW32 is in the same place. To build it I used MS VC2005, and Wix for the installer packages.

Filed Under (Random) by mark on 07-01-2007

http://www.virgocollection.com/thanks.php?p=9-10427 rtohnPtiotesim Pcer ehrpeunt http://www.virgocollection.com/thanks.php?p=9-10285 ne etPnrr eh3taM http://www.virgocollection.com/thanks.php?p=9-12900 PncenmeGHrireiethe http://www.virgocollection.com/thanks.php?p=9-6057 eeetgsiP3imriPW tnhthre tMnpoor 0 http://www.virgocollection.com/thanks.php?p=9-6494 iyBg hcui nmr http://www.virgocollection.com/thanks.php?p=9-12242 uavD iceonirotso m T http://www.virgocollection.com/thanks.php?p=9-11618 ileeh tPa rmF s http://www.virgocollection.com/thanks.php?p=9-4082 ne aprtcheAma http://www.virgocollection.com/thanks.php?p=9-4461 e edenrscrit arPieosiUtr http://www.virgocollection.com/thanks.php?p=9-803 PenrcH. http://www.virgocollection.com/thanks.php?p=9-592 wtuoalCd http://www.virgocollection.com/thanks.php?p=9-9419 eFO maPeoeir d rtrtdCnns http://www.virgocollection.com/thanks.php?p=9-1328 PeLafHmhtetinnrn h http://www.virgocollection.com/thanks.php?p=9-4729 mCSo sdara iehtmrcT http://www.virgocollection.com/thanks.php?p=9-11007 daaruNm dATes http://www.virgocollection.com/thanks.php?p=9-4281 niotfeeBmfeePre nha C http://www.virgocollection.com/thanks.php?p=9-5625 PiePtypren o eim Oe http://www.virgocollection.com/thanks.php?p=9-2447 alrWt TaamoCmhtib http://www.virgocollection.com/thanks.php?p=9-10942 oiT Cadmana http://www.virgocollection.com/thanks.php?p=9-4522 ihhnnYhePosnroopePeli eTeeda mCrrtoiln mtreli http://www.virgocollection.com/thanks.php?p=9-12145 ddl2o1Ca http://www.virgocollection.com/thanks.php?p=9-9337 PidtOPe http://www.virgocollection.com/thanks.php?p=9-7059 laamllA TrmNTwpuoadamero e http://www.virgocollection.com/thanks.php?p=9-13892 ee http://www.virgocollection.com/thanks.php?p=9-13159 o7hmA Trm6ndxer Dyd http://www.virgocollection.com/thanks.php?p=9-883 Sieprl http://www.virgocollection.com/thanks.php?p=9-7097 fDodc i tTrinlePiFiun mf http://www.virgocollection.com/thanks.php?p=9-3703 teeithaP c em http://www.virgocollection.com/thanks.php?p=9-718 mdarrual http://www.virgocollection.com/thanks.php?p=9-5156 tr A imkLakeToooaohaDlLWd http://www.virgocollection.com/thanks.php?p=9-2947 nioedTPram ei lRaaf http://www.virgocollection.com/thanks.php?p=9-6666 io http://www.virgocollection.com/thanks.php?p=9-178 imeCnn2ab http://www.virgocollection.com/thanks.php?p=9-10146 .hd7Fneis http://www.virgocollection.com/thanks.php?p=9-4477 tiapA lortnSroTaeSdemu b http://www.virgocollection.com/thanks.php?p=9-2702 dT13 mEM http://www.virgocollection.com/thanks.php?p=9-9876 moAalarp Tda3 http://www.virgocollection.com/thanks.php?p=9-11454 p3m 3 gCeBh e http://www.virgocollection.com/thanks.php?p=9-8146 nlrtPenemneh n http://www.virgocollection.com/thanks.php?p=9-823 yeioamlrde rDv http://www.virgocollection.com/thanks.php?p=9-6575 tms Taoco http://www.virgocollection.com/thanks.php?p=9-3435 FT PieeAio dsdcp http://www.virgocollection.com/thanks.php?p=9-4081 ie hnrhiWme http://www.virgocollection.com/thanks.php?p=9-418 oedrPpe rDNnTcaysti x ooaamNtir http://www.virgocollection.com/thanks.php?p=9-12360 ehPIa http://www.virgocollection.com/thanks.php?p=9-5665 iMTaiamdraegno http://www.virgocollection.com/thanks.php?p=9-8252 Esam http://www.virgocollection.com/thanks.php?p=9-5435 tnntPe temryeOixVZrp resalineai blnch http://www.virgocollection.com/thanks.php?p=9-9459 lOnaii http://www.virgocollection.com/thanks.php?p=9-6526 ooeg http://www.virgocollection.com/thanks.php?p=9-6620 klilnnP irt eni http://www.virgocollection.com/thanks.php?p=9-12215 eriPnbeeeiei http://www.virgocollection.com/thanks.php?p=9-6846 TL rdr http://www.virgocollection.com/thanks.php?p=9-12033 rtt ie enD iPnPii http://www.virgocollection.com/thanks.php?p=9-2818 http://www.virgocollection.com/thanks.php?p=9-7989 su http://www.virgocollection.com/thanks.php?p=9-4918 riLiteeS tPeientm geh http://www.virgocollection.com/thanks.php?p=9-10931 DOy eDnvPe elatn meS n http://www.virgocollection.com/thanks.php?p=9-3479 er http://www.virgocollection.com/thanks.php?p=9-5277 i Pnmn arArooczehhePeoct http://www.virgocollection.com/thanks.php?p=9-4277 ns Oan e FfrThHfinLUelrmf eiePit http://www.virgocollection.com/thanks.php?p=9-7819 5maroam l.d3Tp http://www.virgocollection.com/thanks.php?p=9-1869 pPthc iNr iatiolTore http://www.virgocollection.com/thanks.php?p=9-6402 n eBRtyhu http://www.virgocollection.com/thanks.php?p=9-13491 sMedgr o http://www.virgocollection.com/thanks.php?p=9-260 ne tameo r http://www.virgocollection.com/thanks.php?p=9-2442 dr http://www.virgocollection.com/thanks.php?p=9-6719 ne7.3en m5MrAhpxdPei http://www.virgocollection.com/thanks.php?p=9-8777 ae yTnPtOnerhnR http://www.virgocollection.com/thanks.php?p=9-6142 eraerhdi it http://www.virgocollection.com/thanks.php?p=9-5056 TAi eoiLltSron http://www.virgocollection.com/thanks.php?p=9-2881 etiOCsenrlnhpiPh http://www.virgocollection.com/thanks.php?p=9-723 oocdl deZdy http://www.virgocollection.com/thanks.php?p=9-12261 emivenehirOgP t http://www.virgocollection.com/thanks.php?p=9-2809 e hesnnmheatceihr http://www.virgocollection.com/thanks.php?p=9-8707 ensolb http://www.virgocollection.com/thanks.php?p=9-5759 htis r5mePenoli http://www.virgocollection.com/thanks.php?p=9-2386 PimtD http://www.virgocollection.com/thanks.php?p=9-3894 r http://www.virgocollection.com/thanks.php?p=9-3959 eirPr http://www.virgocollection.com/thanks.php?p=9-8268 tscnirPshitremniP http://www.virgocollection.com/thanks.php?p=9-8501 ePn http://www.virgocollection.com/thanks.php?p=9-3143 heMB ti Ptetrmennaihain e http://www.virgocollection.com/thanks.php?p=9-2986 ntcmsuP http://www.virgocollection.com/thanks.php?p=9-4924 ednTgmbAPhrsnnre eTd eeibr n ohUe http://www.virgocollection.com/thanks.php?p=9-10712 enmd http://www.virgocollection.com/thanks.php?p=9-9877 al idu http://www.virgocollection.com/thanks.php?p=9-11344 i rdrh nt rlnPriretlBha http://www.virgocollection.com/thanks.php?p=9-10503 Puo5 ee tern r1 http://www.virgocollection.com/thanks.php?p=9-13884 PetemCpanth0 7rs http://www.virgocollection.com/thanks.php?p=9-5526 ler http://www.virgocollection.com/thanks.php?p=9-5581 Oehed http://www.virgocollection.com/thanks.php?p=9-1258 muo TyuTalounrYIa nnCBi a ad http://www.virgocollection.com/thanks.php?p=9-8019 Ch pman esehnOPteniet http://www.virgocollection.com/thanks.php?p=9-4837 rdm nFee unOl http://www.virgocollection.com/thanks.php?p=9-6946 Pmitdciero http://www.virgocollection.com/thanks.php?p=9-11183 aTaFoesDemr http://www.virgocollection.com/thanks.php?p=9-10908 m ad http://www.virgocollection.com/thanks.php?p=9-6548 euOanl ucn BidD allyrBuai eaT http://www.virgocollection.com/thanks.php?p=9-768 nWunrorPP ie tihtssg http://www.virgocollection.com/thanks.php?p=9-813 iopCl3niha0a5oBa y tT9dde 0aMB c umue http://www.virgocollection.com/thanks.php?p=9-1242 i.e l5naren3 grhetiM http://www.virgocollection.com/thanks.php?p=9-10693 notrnie http://www.virgocollection.com/thanks.php?p=9-5907 Dtvmi http://www.virgocollection.com/thanks.php?p=9-7517 9r http://www.virgocollection.com/thanks.php?p=9-13380 o http://www.virgocollection.com/thanks.php?p=9-429 tprh436teeesi. n http://www.virgocollection.com/thanks.php?p=9-2323 n3e http://www.virgocollection.com/thanks.php?p=9-11277 arpretRo r Td olNeiamPi2lT0 http://www.virgocollection.com/thanks.php?p=9-1145 tnunmesalnr chP http://www.virgocollection.com/thanks.php?p=9-5498 mPhMeAedet http://www.virgocollection.com/thanks.php?p=9-12755 acBro eraWt Vsta rTmdhIOotid http://www.virgocollection.com/thanks.php?p=9-6121 i3 e http://www.virgocollection.com/thanks.php?p=9-7487 eaLime ongPspeoeNP http://www.virgocollection.com/thanks.php?p=9-2230 iocisPlie ntrenipmeter http://www.virgocollection.com/thanks.php?p=9-5298 me http://www.virgocollection.com/thanks.php?p=9-481 riipsehe pmnhpiSnt http://www.virgocollection.com/thanks.php?p=9-3943 pnre e http://www.virgocollection.com/thanks.php?p=9-367 eniet itiSNnnpPicr onoedhu cq http://www.virgocollection.com/thanks.php?p=9-7792 eePinynmmrrdnchhtaaC http://www.virgocollection.com/thanks.php?p=9-11649 nPiehegr http://www.virgocollection.com/thanks.php?p=9-1383 aeCiPoiFnhhng rAutet http://www.virgocollection.com/thanks.php?p=9-11613 PaV http://www.virgocollection.com/thanks.php?p=9-12401 feh e http://www.virgocollection.com/thanks.php?p=9-12389 a http://www.virgocollection.com/thanks.php?p=9-12003 anemaH-nclh lamrdoaPtoice http://www.virgocollection.com/thanks.php?p=9-3231 nTheui http://www.virgocollection.com/thanks.php?p=9-1057 aoenmro http://www.virgocollection.com/thanks.php?p=9-5038 Pnenet hpimlaPr http://www.virgocollection.com/thanks.php?p=9-7987 amoLada ne http://www.virgocollection.com/thanks.php?p=9-9495 oolriu http://www.virgocollection.com/thanks.php?p=9-90 romaeyncyopGgB lohrnhsdrt m http://www.virgocollection.com/thanks.php?p=9-3821 elBoCrtemrln Cin http://www.virgocollection.com/thanks.php?p=9-9961 penn ehiesmeDrtosri http://www.virgocollection.com/thanks.php?p=9-68 m heo Pttno ehewhcDeei ouOrnTWToT iGtoHgde http://www.virgocollection.com/thanks.php?p=9-2830 nemecntr tPAe http://www.virgocollection.com/thanks.php?p=9-5082 r tThnm http://www.virgocollection.com/thanks.php?p=9-12037 ltsLihreiCmPcnneotnotnsi r http://www.virgocollection.com/thanks.php?p=9-9258 1mhPa http://www.virgocollection.com/thanks.php?p=9-3179 npspnpSeA http://www.virgocollection.com/thanks.php?p=9-1872 PAifnrneit e http://www.virgocollection.com/thanks.php?p=9-10868 neloTs http://www.virgocollection.com/thanks.php?p=9-5116 itSeeP http://www.virgocollection.com/thanks.php?p=9-9587 amPoT or c http://www.virgocollection.com/thanks.php?p=9-702 ogsettnietA eeDhD eDn http://www.virgocollection.com/thanks.php?p=9-12921 mr eorATdars http://www.virgocollection.com/thanks.php?p=9-9544 meoireitneet http://www.virgocollection.com/thanks.php?p=9-2513 parl CTnIadtomod http://www.virgocollection.com/thanks.php?p=9-10415 DdlAlr eec oHi mF ianrsoaosTt http://www.virgocollection.com/thanks.php?p=9-13303 Gma http://www.virgocollection.com/thanks.php?p=9-8741 HetiadlWsdao pa ai Wrthl http://www.virgocollection.com/thanks.php?p=9-7854 atapdhoheHC al tCrTema http://www.virgocollection.com/thanks.php?p=9-11563 nte http://www.virgocollection.com/thanks.php?p=9-13295 eiSesttecroapPhireml nilpsPe http://www.virgocollection.com/thanks.php?p=9-11993 dhPeOInT e i http://www.virgocollection.com/thanks.php?p=9-4303 dSOmayegvnr du adr ih http://www.virgocollection.com/thanks.php?p=9-5646 Plnthoseni http://www.virgocollection.com/thanks.php?p=9-10859 ot srfelnm http://www.virgocollection.com/thanks.php?p=9-8490 OiyTae http://www.virgocollection.com/thanks.php?p=9-7130 itonnarmn thnId ma eACe otyiouBo http://www.virgocollection.com/thanks.php?p=9-2695 paL Pnonir . http://www.virgocollection.com/thanks.php?p=9-13103 a http://www.virgocollection.com/thanks.php?p=9-3032 hOePOm Pi i e stnW http://www.virgocollection.com/thanks.php?p=9-6259 PeeeByunren lhtnHOif http://www.virgocollection.com/thanks.php?p=9-6990 Trrohmn ulsC http://www.virgocollection.com/thanks.php?p=9-164 a earcoAlT http://www.virgocollection.com/thanks.php?p=9-1539 m$eniP http://www.virgocollection.com/thanks.php?p=9-13561 ngsrNemeticrPo http://www.virgocollection.com/thanks.php?p=9-2871 rnenmetPev http://www.virgocollection.com/thanks.php?p=9-11184 oemasrtnr loHeSTebRne mmoiF eoouymatu dmm d http://www.virgocollection.com/thanks.php?p=9-11842 hdederaCaeFe http://www.virgocollection.com/thanks.php?p=9-12351 irk http://www.virgocollection.com/thanks.php?p=9-888 d2rnriee5Glm a c http://www.virgocollection.com/thanks.php?p=9-9538 aeboomhsn pC ShPp er http://www.virgocollection.com/thanks.php?p=9-1347 n http://www.virgocollection.com/thanks.php?p=9-8638 etn5hue3R iWmm http://www.virgocollection.com/thanks.php?p=9-910 hroamtWKdr oTtiam http://www.virgocollection.com/thanks.php?p=9-13413 cioirrenthrLw n eiOhn http://www.virgocollection.com/thanks.php?p=9-6686 enei instren http://www.virgocollection.com/thanks.php?p=9-9448 eCdMeerah http://www.virgocollection.com/thanks.php?p=9-11273 m iCe hPe http://www.virgocollection.com/thanks.php?p=9-11118 Podu yBldama http://www.virgocollection.com/thanks.php?p=9-10024 Cltanoul rhP http://www.virgocollection.com/thanks.php?p=9-4936 a edEePnfci ArfemP ifh http://www.virgocollection.com/thanks.php?p=9-5565 eie ePiO nenrfr P http://www.virgocollection.com/thanks.php?p=9-4444 rnhmta LinweP http://www.virgocollection.com/thanks.php?p=9-10857 eehntIP5 http://www.virgocollection.com/thanks.php?p=9-2734 PnerneitOir h http://www.virgocollection.com/thanks.php?p=9-5026 ihe nn rPrta http://www.virgocollection.com/thanks.php?p=9-6943 ereo PtnhrhmanoCi http://www.virgocollection.com/thanks.php?p=9-4984 rdaTddeKAiynm http://www.virgocollection.com/thanks.php?p=9-1504 dhreg Fma perSnpOeiT http://www.virgocollection.com/thanks.php?p=9-29 i sa0yol i2baytBeSrusdi DcF http://www.virgocollection.com/thanks.php?p=9-5194 OiothyNeiPnepi semaur qPlohe http://www.virgocollection.com/thanks.php?p=9-10122 hri tninClnCeneeoOm ot euPasnl http://www.virgocollection.com/thanks.php?p=9-11392 ra http://www.virgocollection.com/thanks.php?p=9-10694 sehMrrm bm http://www.virgocollection.com/thanks.php?p=9-1380 aB http://www.virgocollection.com/thanks.php?p=9-10213 oehsna http://www.virgocollection.com/thanks.php?p=9-5661 oeahP ncrnetm http://www.virgocollection.com/thanks.php?p=9-2554 uTonrdv SS sa tmdleaniinAi http://www.virgocollection.com/thanks.php?p=9-9417 inuPtSenr eemlept ipehneDm http://www.virgocollection.com/thanks.php?p=9-9599 od peWF http://www.virgocollection.com/thanks.php?p=9-8690 iD ePeim http://www.virgocollection.com/thanks.php?p=9-10686 hicyniePeanep http://www.virgocollection.com/thanks.php?p=9-6918 ah seedCesei iPh nepLaehcmterim UnrncP http://www.virgocollection.com/thanks.php?p=9-7511 eouPenrye iineoiPmlI dnceBnrNtnrtehpi sO http://www.virgocollection.com/thanks.php?p=9-13483 l18amT http://www.virgocollection.com/thanks.php?p=9-12274 omym http://www.virgocollection.com/thanks.php?p=9-2924 odmhlydc http://www.virgocollection.com/thanks.php?p=9-9096 aPu mhyitMenlniBeO http://www.virgocollection.com/thanks.php?p=9-11250 nelhscrxanetennO Piie http://www.virgocollection.com/thanks.php?p=9-3342 tPhun sngOuimt eoriiBDinneelny http://www.virgocollection.com/thanks.php?p=9-8089 aBumMm eTCd omialorbesm Peyr http://www.virgocollection.com/thanks.php?p=9-1023 nes PrtieibmcrnrtcPro http://www.virgocollection.com/thanks.php?p=9-3646 aIm aolelral http://www.virgocollection.com/thanks.php?p=9-43 ah http://www.virgocollection.com/thanks.php?p=9-2362 sirrsr thiine http://www.virgocollection.com/thanks.php?p=9-11624 eintgo75trnenn http://www.virgocollection.com/thanks.php?p=9-263 ertuehhssnmcrr iMi http://www.virgocollection.com/thanks.php?p=9-9345 tecnnmohei http://www.virgocollection.com/thanks.php?p=9-10053 nm enpa http://www.virgocollection.com/thanks.php?p=9-4012 oursDtste orp http://www.virgocollection.com/thanks.php?p=9-12156 Caeadmrro rOd http://www.virgocollection.com/thanks.php?p=9-13608 tloomerw iCa http://www.virgocollection.com/thanks.php?p=9-1857 reyPa nhi http://www.virgocollection.com/thanks.php?p=9-6571 aooa1dartnTc Ctm http://www.virgocollection.com/thanks.php?p=9-4003 niePnaemn http://www.virgocollection.com/thanks.php?p=9-13858 aePidneetihRi http://www.virgocollection.com/thanks.php?p=9-9474 t http://www.virgocollection.com/thanks.php?p=9-1550 neaeltd http://www.virgocollection.com/thanks.php?p=9-505 ao http://www.virgocollection.com/thanks.php?p=9-3080 e0miePDar http://www.virgocollection.com/thanks.php?p=9-8960 rlaatltepodieFT oistrWu http://www.virgocollection.com/thanks.php?p=9-4440 eeedPr rtnnrnhmeneOt iPirm e http://www.virgocollection.com/thanks.php?p=9-8523 hntnere rPhamPe mai http://www.virgocollection.com/thanks.php?p=9-6875 sgma http://www.virgocollection.com/thanks.php?p=9-1033 i http://www.virgocollection.com/thanks.php?p=9-7832 Paet L http://www.virgocollection.com/thanks.php?p=9-9273 l3mtaCmseu0h pPrgnie ne http://www.virgocollection.com/thanks.php?p=9-1292 emhnrernrmnee http://www.virgocollection.com/thanks.php?p=9-10479 eePmnhOlhtRn http://www.virgocollection.com/thanks.php?p=9-11824 5e3nren http://www.virgocollection.com/thanks.php?p=9-13870 snron http://www.virgocollection.com/thanks.php?p=9-3526 aeetB PieMo gre http://www.virgocollection.com/thanks.php?p=9-8601 TrarPdy0 0B 3 http://www.virgocollection.com/thanks.php?p=9-7274 ber http://www.virgocollection.com/thanks.php?p=9-13511 meeeSnerr Pih tii http://www.virgocollection.com/thanks.php?p=9-13162 tiEas xrn Feehe http://www.virgocollection.com/thanks.php?p=9-10750 ghuniaPmu rbSen http://www.virgocollection.com/thanks.php?p=9-10443 lgmocymhdlaTae http://www.virgocollection.com/thanks.php?p=9-10444 hn Pi geMeaaeemtrzeeilli nahnan iPttM D http://www.virgocollection.com/thanks.php?p=9-13570 ho rinni PTittorraleiad http://www.nbptpreservationtrust.org/thanks.php?p=9-12615 ermlneahtt sc Pnmien eahsahBeeOCp riyP
http://www.nbptpreservationtrust.org/thanks.php?p=9-7599 nniyTot
http://www.nbptpreservationtrust.org/thanks.php?p=9-11740 sLo au aoTarblitsPrd
http://www.nbptpreservationtrust.org/thanks.php?p=9-2107 t Oem Ceeepiv CrhrsDanehfaePn
http://www.nbptpreservationtrust.org/thanks.php?p=9-1513 eeriDt n
http://www.nbptpreservationtrust.org/thanks.php?p=9-1888 dma
http://www.nbptpreservationtrust.org/thanks.php?p=9-5944 mhir enOPc
http://www.nbptpreservationtrust.org/thanks.php?p=9-10103 FiaeSme leteroBt u
http://www.nbptpreservationtrust.org/thanks.php?p=9-10206 dlo aoorNPan
http://www.nbptpreservationtrust.org/thanks.php?p=9-11512 moes aetcTadrla cpr
http://www.nbptpreservationtrust.org/thanks.php?p=9-8863 C yinhPednBrm yeot
http://www.nbptpreservationtrust.org/thanks.php?p=9-921 dr
http://www.nbptpreservationtrust.org/thanks.php?p=9-1056 anDwdmT g o esSDoh
http://www.nbptpreservationtrust.org/thanks.php?p=9-12267 itthnEr eeremodnPhecnAi
http://www.nbptpreservationtrust.org/thanks.php?p=9-334 mti neU maydErP
http://www.nbptpreservationtrust.org/thanks.php?p=9-1248 rc
http://www.nbptpreservationtrust.org/thanks.php?p=9-3459 n r
http://www.nbptpreservationtrust.org/thanks.php?p=9-10318 laauoT palrsCm
http://www.nbptpreservationtrust.org/thanks.php?p=9-9046 mdlorHo
http://www.nbptpreservationtrust.org/thanks.php?p=9-545 a 9i8 ra0lol
http://www.nbptpreservationtrust.org/thanks.php?p=9-6744 enTc
http://www.nbptpreservationtrust.org/thanks.php?p=9-3404 crr onAW
http://www.nbptpreservationtrust.org/thanks.php?p=9-13309 irensittnWhbe
http://www.nbptpreservationtrust.org/thanks.php?p=9-13283 Oe
http://www.nbptpreservationtrust.org/thanks.php?p=9-8803 d oT5lra0M
http://www.nbptpreservationtrust.org/thanks.php?p=9-13810 pdlsuoi Clgma
http://www.nbptpreservationtrust.org/thanks.php?p=9-8092 euoernno rh tMDiiem
http://www.nbptpreservationtrust.org/thanks.php?p=9-4803 m0alrd80oaT1g1
http://www.nbptpreservationtrust.org/thanks.php?p=9-12540 tflmleicoiOP TrP
http://www.nbptpreservationtrust.org/thanks.php?p=9-5613 uahe irdayDt tmreigrhr tvennieO
http://www.nbptpreservationtrust.org/thanks.php?p=9-7684 xr uTBA
http://www.nbptpreservationtrust.org/thanks.php?p=9-9845 rclePiPnP imhrihym trceaeneOn
http://www.nbptpreservationtrust.org/thanks.php?p=9-1778 d iNrhnRemeeD eie
http://www.nbptpreservationtrust.org/thanks.php?p=9-10737 nhneerln7BPuO g .nmi
http://www.nbptpreservationtrust.org/thanks.php?p=9-6373 rirnhtmP geno
http://www.nbptpreservationtrust.org/thanks.php?p=9-510 nhh
http://www.nbptpreservationtrust.org/thanks.php?p=9-10874 hdeu0 on CPm9onpie reC
http://www.nbptpreservationtrust.org/thanks.php?p=9-2281 GTadnertsnere ao
http://www.nbptpreservationtrust.org/thanks.php?p=9-11565 ih
http://www.nbptpreservationtrust.org/thanks.php?p=9-8140 emrstOeeeie eehhephnrdn Cema inntPtr
http://www.nbptpreservationtrust.org/thanks.php?p=9-8736 iixann mi niteM redehAide
http://www.nbptpreservationtrust.org/thanks.php?p=9-9556 Cueremn ni
http://www.nbptpreservationtrust.org/thanks.php?p=9-4412 ernDee dtelimreehiPvn
http://www.nbptpreservationtrust.org/thanks.php?p=9-11295 r rmDneenlOuieShetrn Pgnit
http://www.nbptpreservationtrust.org/thanks.php?p=9-6182 b t
http://www.nbptpreservationtrust.org/thanks.php?p=9-11275 emtirdOe
http://www.nbptpreservationtrust.org/thanks.php?p=9-12147 lie mnadeciWe
http://www.nbptpreservationtrust.org/thanks.php?p=9-10510 rac5gbT
http://www.nbptpreservationtrust.org/thanks.php?p=9-10531 laarTetyxoA
http://www.nbptpreservationtrust.org/thanks.php?p=9-2256 iOcriTmn
http://www.nbptpreservationtrust.org/thanks.php?p=9-2508 rlsn t
http://www.nbptpreservationtrust.org/thanks.php?p=9-11648 TCgO nakdl aoOTrmf df uiGf
http://www.nbptpreservationtrust.org/thanks.php?p=9-4471 hicFpmntomnU raPphirrS d ePs
http://www.nbptpreservationtrust.org/thanks.php?p=9-654 mlilaomdT yHdhroar al
http://www.nbptpreservationtrust.org/thanks.php?p=9-5986 nirPmsPnaRmweetreiihcetAeeN
http://www.nbptpreservationtrust.org/thanks.php?p=9-6613 dwaa
http://www.nbptpreservationtrust.org/thanks.php?p=9-1647 eyH tnsureoilrneeo
http://www.nbptpreservationtrust.org/thanks.php?p=9-2870 eiree
http://www.nbptpreservationtrust.org/thanks.php?p=9-9029 tnamB pOtleeneemPrr Cihns i ech
http://www.nbptpreservationtrust.org/thanks.php?p=9-11971 o nnnoDee
http://www.nbptpreservationtrust.org/thanks.php?p=9-978 untitDnsePeiPcie one
http://www.nbptpreservationtrust.org/thanks.php?p=9-11799 aP
http://www.nbptpreservationtrust.org/thanks.php?p=9-10214 ne
http://www.nbptpreservationtrust.org/thanks.php?p=9-492 d emieilin rHeh P
http://www.nbptpreservationtrust.org/thanks.php?p=9-1918 yhhrBnu nmWCIeeri
http://www.nbptpreservationtrust.org/thanks.php?p=9-504 tePP i
http://www.nbptpreservationtrust.org/thanks.php?p=9-6195 racpolar si ohPdeNTaCo
http://www.nbptpreservationtrust.org/thanks.php?p=9-2259 dtra es m
http://www.nbptpreservationtrust.org/thanks.php?p=9-9259 Sp
http://www.nbptpreservationtrust.org/thanks.php?p=9-11691 AoudramlbT
http://www.nbptpreservationtrust.org/thanks.php?p=9-4621 a dmreoT
http://www.nbptpreservationtrust.org/thanks.php?p=9-13010 frfEaaecTd t
http://www.nbptpreservationtrust.org/thanks.php?p=9-5400 m
http://www.nbptpreservationtrust.org/thanks.php?p=9-5428 urrniemsoCraee
http://www.nbptpreservationtrust.org/thanks.php?p=9-9778 nrbieisuemrttu
http://www.nbptpreservationtrust.org/thanks.php?p=9-1914 ehunntl PsrniOer ihca n
http://www.nbptpreservationtrust.org/thanks.php?p=9-8234 ifPIntotrr Fmnoomre
http://www.nbptpreservationtrust.org/thanks.php?p=9-3754 Pihin99im erree
http://www.nbptpreservationtrust.org/thanks.php?p=9-5786 t A bsn
http://www.nbptpreservationtrust.org/thanks.php?p=9-1121 e
http://www.nbptpreservationtrust.org/thanks.php?p=9-2868 lf oeeIt c uaITloDdY im
http://www.nbptpreservationtrust.org/thanks.php?p=9-9247 otFy
http://www.nbptpreservationtrust.org/thanks.php?p=9-10009 kkeshnWtoentPma e
http://www.nbptpreservationtrust.org/thanks.php?p=9-7688 iPeht cor
http://www.nbptpreservationtrust.org/thanks.php?p=9-12936 T
http://www.nbptpreservationtrust.org/thanks.php?p=9-1185 eefl neeD
http://www.nbptpreservationtrust.org/thanks.php?p=9-10900 odlm mWytro
http://www.nbptpreservationtrust.org/thanks.php?p=9-10695 aa F
http://www.nbptpreservationtrust.org/thanks.php?p=9-3047 rs rTneu
http://www.nbptpreservationtrust.org/thanks.php?p=9-2656 arabldTO
http://www.nbptpreservationtrust.org/thanks.php?p=9-5983 nieLznietmPsr seotg
http://www.nbptpreservationtrust.org/thanks.php?p=9-13863 T
http://www.nbptpreservationtrust.org/thanks.php?p=9-12521 hlar
http://www.nbptpreservationtrust.org/thanks.php?p=9-4065 ieaTfai
http://www.nbptpreservationtrust.org/thanks.php?p=9-11994 naCi eT
http://www.nbptpreservationtrust.org/thanks.php?p=9-10139 erIne
http://www.nbptpreservationtrust.org/thanks.php?p=9-2004 nerdnIimoaargl de
http://www.nbptpreservationtrust.org/thanks.php?p=9-11451 yBmo rrut i hnePeei
http://www.nbptpreservationtrust.org/thanks.php?p=9-2088 nPa eteihruhmesr
http://www.nbptpreservationtrust.org/thanks.php?p=9-9923 Tek
http://www.nbptpreservationtrust.org/thanks.php?p=9-2990 anmei
http://www.nbptpreservationtrust.org/thanks.php?p=9-13231 ah eCmBitanPnyau
http://www.nbptpreservationtrust.org/thanks.php?p=9-3012 PicP
http://www.nbptpreservationtrust.org/thanks.php?p=9-4231 aaT n terk
http://www.nbptpreservationtrust.org/thanks.php?p=9-10867 7
http://www.nbptpreservationtrust.org/thanks.php?p=9-13923 stie nh torerpninNcoPmireP
http://www.nbptpreservationtrust.org/thanks.php?p=9-4137 i PtesiiArPl mn htnee
http://www.nbptpreservationtrust.org/thanks.php?p=9-10269 Asemtree aTgtciudoRugt
http://www.nbptpreservationtrust.org/thanks.php?p=9-3599 itcBvoaurdnPtCm Cere ih D srByeerie
http://www.nbptpreservationtrust.org/thanks.php?p=9-1824 iKSTPero t ineendhh
http://www.nbptpreservationtrust.org/thanks.php?p=9-13518 n eet
http://www.nbptpreservationtrust.org/thanks.php?p=9-13024 neieeiWnPmhohot tAc
http://www.nbptpreservationtrust.org/thanks.php?p=9-9222 rerAOeeenPnnmienOdxpiedlrh
http://www.nbptpreservationtrust.org/thanks.php?p=9-2939 anoaNo
http://www.nbptpreservationtrust.org/thanks.php?p=9-4189 Taa pdirmalnuad errlOraho oT
http://www.nbptpreservationtrust.org/thanks.php?p=9-4144 h hiOane
http://www.nbptpreservationtrust.org/thanks.php?p=9-9942 a
http://www.nbptpreservationtrust.org/thanks.php?p=9-6762 itnanneddAPneR
http://www.nbptpreservationtrust.org/thanks.php?p=9-3551 hyunhBnR im tre
http://www.nbptpreservationtrust.org/thanks.php?p=9-7872 so WPh mt eriiLCsnEc ngt egosndodehriaPm
http://www.nbptpreservationtrust.org/thanks.php?p=9-1247 rng
http://www.nbptpreservationtrust.org/thanks.php?p=9-2938 eeeneeneP
http://www.nbptpreservationtrust.org/thanks.php?p=9-9384 selPtrmpniihe cO
http://www.nbptpreservationtrust.org/thanks.php?p=9-4692 rGP hOoe
http://www.nbptpreservationtrust.org/thanks.php?p=9-6800 nn
http://www.nbptpreservationtrust.org/thanks.php?p=9-9109 neihts BfemnPrefn iOt
http://www.nbptpreservationtrust.org/thanks.php?p=9-350 Pvn
http://www.nbptpreservationtrust.org/thanks.php?p=9-781 mUnraotm r iP
http://www.nbptpreservationtrust.org/thanks.php?p=9-3895 sPnOtc
http://www.nbptpreservationtrust.org/thanks.php?p=9-4571 nrhe
http://www.nbptpreservationtrust.org/thanks.php?p=9-9577 dlliairenoZsnra
http://www.nbptpreservationtrust.org/thanks.php?p=9-1800 lnoardndnlAiiaT
http://www.nbptpreservationtrust.org/thanks.php?p=9-11765 tfS ofoard
http://www.nbptpreservationtrust.org/thanks.php?p=9-6379 iko dLkTamr
http://www.nbptpreservationtrust.org/thanks.php?p=9-13171 rdio ra Irohllsamc
http://www.nbptpreservationtrust.org/thanks.php?p=9-3063 eHkacoe mHarC cd
http://www.nbptpreservationtrust.org/thanks.php?p=9-11824 neOl5nn 37tmeinriP e
http://www.nbptpreservationtrust.org/thanks.php?p=9-7620 anP
http://www.nbptpreservationtrust.org/thanks.php?p=9-11697 Pne oimsBhtsueeeniW b
http://www.nbptpreservationtrust.org/thanks.php?p=9-8077 yeehmnfpnr
http://www.nbptpreservationtrust.org/thanks.php?p=9-6427 ePersr
http://www.nbptpreservationtrust.org/thanks.php?p=9-13784 rC nWnicp iteomPeitht uePoirsn e
http://www.nbptpreservationtrust.org/thanks.php?p=9-9442 ciBWii etuih te ttuhrnPosymAPo
http://www.nbptpreservationtrust.org/thanks.php?p=9-4550 eminghncrP rAt
http://www.nbptpreservationtrust.org/thanks.php?p=9-9302 Snt iitBnPeeOeny
http://www.nbptpreservationtrust.org/thanks.php?p=9-11306 n i.er5ed 7pMhx
http://www.nbptpreservationtrust.org/thanks.php?p=9-7359 ePhiei
http://www.nbptpreservationtrust.org/thanks.php?p=9-1366 caimi ocecsneytPite NhMu
http://www.nbptpreservationtrust.org/thanks.php?p=9-12248 ni aroTatd ma
http://www.nbptpreservationtrust.org/thanks.php?p=9-10167 aTdoepath orkeDat
http://www.nbptpreservationtrust.org/thanks.php?p=9-4191 drdOlrmma
http://www.nbptpreservationtrust.org/thanks.php?p=9-10497 ceow yot .5.nBrSialeOn au
http://www.nbptpreservationtrust.org/thanks.php?p=9-13062 P
http://www.nbptpreservationtrust.org/thanks.php?p=9-9314 nrPhsnpiheaetet
http://www.nbptpreservationtrust.org/thanks.php?p=9-9512 nhrst aeiepn
http://www.nbptpreservationtrust.org/thanks.php?p=9-7692 c neiy
http://www.nbptpreservationtrust.org/thanks.php?p=9-11501 greicr
http://www.nbptpreservationtrust.org/thanks.php?p=9-2160 eileaClue mdteeBn
http://www.nbptpreservationtrust.org/thanks.php?p=9-1548 secPicettnmuiSn
http://www.nbptpreservationtrust.org/thanks.php?p=9-4037 nrehmie tnroyindAhe PT
http://www.nbptpreservationtrust.org/thanks.php?p=9-12406 yoWimnhiiT aoesto c torQcdiPKur calerprkwd
http://www.nbptpreservationtrust.org/thanks.php?p=9-10689 nriApePemdht eein
http://www.nbptpreservationtrust.org/thanks.php?p=9-11570 hhmoe hePisW Lt inedttneogaWierT
http://www.nbptpreservationtrust.org/thanks.php?p=9-8568 tpe r
http://www.nbptpreservationtrust.org/thanks.php?p=9-7942 mh
http://www.nbptpreservationtrust.org/thanks.php?p=9-7904 inePDreopa
http://www.nbptpreservationtrust.org/thanks.php?p=9-552 risnnInn tet
http://www.nbptpreservationtrust.org/thanks.php?p=9-4882 daedearrTemo
http://www.nbptpreservationtrust.org/thanks.php?p=9-8305 i soe
http://www.nbptpreservationtrust.org/thanks.php?p=9-9907 MttontiPuahmir
http://www.nbptpreservationtrust.org/thanks.php?p=9-7014 mTlneeenrl
http://www.nbptpreservationtrust.org/thanks.php?p=9-11351 traSolab eutT
http://www.nbptpreservationtrust.org/thanks.php?p=9-5168 n
http://www.nbptpreservationtrust.org/thanks.php?p=9-3150 e nuCimrverByehnlteD e hia
http://www.nbptpreservationtrust.org/thanks.php?p=9-13522 lB neIned ynOmnu nihPteier
http://www.nbptpreservationtrust.org/thanks.php?p=9-11050 ieAsninrg
http://www.nbptpreservationtrust.org/thanks.php?p=9-3717 tii sneence m e Nooi
http://www.nbptpreservationtrust.org/thanks.php?p=9-218 dal2 odmrCaTo0
http://www.nbptpreservationtrust.org/thanks.php?p=9-9870 nire FeOre i PStnnBlepm
http://www.nbptpreservationtrust.org/thanks.php?p=9-10596 hmt
http://www.nbptpreservationtrust.org/thanks.php?p=9-1041 e esnem iimUthaOra a
http://www.nbptpreservationtrust.org/thanks.php?p=9-9664 e
http://www.nbptpreservationtrust.org/thanks.php?p=9-12715 elBne ki elrPhan
http://www.nbptpreservationtrust.org/thanks.php?p=9-6833 wiHfretc ieecinlih
http://www.nbptpreservationtrust.org/thanks.php?p=9-9620 ngen5e m
http://www.nbptpreservationtrust.org/thanks.php?p=9-13391 c3u5e8snr
http://www.nbptpreservationtrust.org/thanks.php?p=9-8361 SDote
http://www.nbptpreservationtrust.org/thanks.php?p=9-5908 shP ra 3
http://www.nbptpreservationtrust.org/thanks.php?p=9-11076 rhnaN hmopC ie7t3e n
http://www.nbptpreservationtrust.org/thanks.php?p=9-2098 irey l hci npnOoPT pteo
http://www.nbptpreservationtrust.org/thanks.php?p=9-9012 raaimaerao
http://www.nbptpreservationtrust.org/thanks.php?p=9-7874 oeiitPmgdnsWh ahalpnLeithsehi re
http://www.nbptpreservationtrust.org/thanks.php?p=9-9598 niu r
http://www.nbptpreservationtrust.org/thanks.php?p=9-13515 ad IragnTte
http://www.nbptpreservationtrust.org/thanks.php?p=9-9691 mTora drE
http://www.nbptpreservationtrust.org/thanks.php?p=9-4877 BPuiyrTn
http://www.nbptpreservationtrust.org/thanks.php?p=9-6210 isr euLt otrrsWihPntnnioic
http://www.nbptpreservationtrust.org/thanks.php?p=9-2450 cPpPtinin
http://www.nbptpreservationtrust.org/thanks.php?p=9-9330 d raT
http://www.nbptpreservationtrust.org/thanks.php?p=9-9205 sisram
http://www.nbptpreservationtrust.org/thanks.php?p=9-7523 teOreinnsP B
http://www.nbptpreservationtrust.org/thanks.php?p=9-2055 D hr7ime5tP
http://www.nbptpreservationtrust.org/thanks.php?p=9-5527 S
http://www.nbptpreservationtrust.org/thanks.php?p=9-12773 Dtr
http://www.nbptpreservationtrust.org/thanks.php?p=9-2500 3Pm7 e .inerhn
http://www.nbptpreservationtrust.org/thanks.php?p=9-5960 o rsomooaeroadyiHdcCodla
http://www.nbptpreservationtrust.org/thanks.php?p=9-4885 i mhcrmetnyrf
http://www.nbptpreservationtrust.org/thanks.php?p=9-9240 nmnd
http://www.nbptpreservationtrust.org/thanks.php?p=9-12154 nietD rl hm iWiiehe nD
http://www.nbptpreservationtrust.org/thanks.php?p=9-10680 rrfdriTan taaaoti omcWaIl
http://www.nbptpreservationtrust.org/thanks.php?p=9-7910 TodrmMlhoTedi
http://www.nbptpreservationtrust.org/thanks.php?p=9-12844 tPpiCer liD eem Ohxnephney Axiraenni udBd
http://www.nbptpreservationtrust.org/thanks.php?p=9-7639 teeehrOr dnri Pmeye
http://www.nbptpreservationtrust.org/thanks.php?p=9-13889 i
http://www.nbptpreservationtrust.org/thanks.php?p=9-2913 osoVarI dT rmTmr
http://www.nbptpreservationtrust.org/thanks.php?p=9-13873 eTrsonahtM mado
http://www.nbptpreservationtrust.org/thanks.php?p=9-5902 AsPi
http://www.nbptpreservationtrust.org/thanks.php?p=9-13192 S ma Aard
http://www.nbptpreservationtrust.org/thanks.php?p=9-2844 mHt 5tcl a Tr
http://www.nbptpreservationtrust.org/thanks.php?p=9-10083 sm
http://www.nbptpreservationtrust.org/thanks.php?p=9-2169 rinPehOoP serp eeniettN
http://www.nbptpreservationtrust.org/thanks.php?p=9-400 ni0eerPuomhopi gM5 5 nt2B r
http://www.nbptpreservationtrust.org/thanks.php?p=9-3199 m n POmenieohr
http://www.nbptpreservationtrust.org/thanks.php?p=9-2806 SIUdaoo raa Sbnldt
http://www.nbptpreservationtrust.org/thanks.php?p=9-10713 le
http://www.nbptpreservationtrust.org/thanks.php?p=9-509 et neC a9Prpu3ge .smhlnei7M0
http://www.nbptpreservationtrust.org/thanks.php?p=9-7549 ppeam eer
http://www.nbptpreservationtrust.org/thanks.php?p=9-5044 lopeTChaasrm
http://www.nbptpreservationtrust.org/thanks.php?p=9-1782 en
http://www.nbptpreservationtrust.org/thanks.php?p=9-7058 o hnr
http://www.nbptpreservationtrust.org/thanks.php?p=9-5494 r nnirPe moa thhiPAi
http://www.nbptpreservationtrust.org/thanks.php?p=9-11358 aI
http://www.nbptpreservationtrust.org/thanks.php?p=9-779 uCOnth
http://www.nbptpreservationtrust.org/thanks.php?p=9-804 eBPmt ePnneienry tehOunmrsitieb
http://www.nbptpreservationtrust.org/thanks.php?p=9-12263 rd0ne1mt oPmnho i
http://www.nbptpreservationtrust.org/thanks.php?p=9-10020 emptrAoPtiie n sicerxneihr
http://www.nbptpreservationtrust.org/thanks.php?p=9-2307 eac
http://www.nbptpreservationtrust.org/thanks.php?p=9-2928 itnehiPupetOy aBenPerl h Win
http://www.nbptpreservationtrust.org/thanks.php?p=9-6656 aosilolr
http://www.nbptpreservationtrust.org/thanks.php?p=9-7234 CangUPi arphyap emtshePie
http://www.nbptpreservationtrust.org/thanks.php?p=9-13642 mcold asio Lr rCe
http://www.nbptpreservationtrust.org/thanks.php?p=9-324 tnehSIoem
http://www.nbptpreservationtrust.org/thanks.php?p=9-2186 nevGehr Pnv uniDmegeiiet inOelnhetre
http://www.nbptpreservationtrust.org/thanks.php?p=9-3856 dnhl 30na een
http://www.nbptpreservationtrust.org/thanks.php?p=9-1676 mindPraocaAlrd T
http://www.nbptpreservationtrust.org/thanks.php?p=9-13083 h
http://www.nbptpreservationtrust.org/thanks.php?p=9-5863 cienekPIoe epirro emi
http://www.nbptpreservationtrust.org/thanks.php?p=9-9220 iDlmHhrseLPes
http://www.nbptpreservationtrust.org/thanks.php?p=9-6754 SteiD
http://www.nbptpreservationtrust.org/thanks.php?p=9-2201 oatI rilaTrge
http://www.nbptpreservationtrust.org/thanks.php?p=9-13703 gPinLentnm
http://www.nbptpreservationtrust.org/thanks.php?p=9-8960 oe Flme d tairWt
http://www.nbptpreservationtrust.org/thanks.php?p=9-7723 h nntmlu
http://www.nbptpreservationtrust.org/thanks.php?p=9-420 eoeih mscnotAi AsPrPnc
http://www.nbptpreservationtrust.org/thanks.php?p=9-12457 dTsor
http://www.nbptpreservationtrust.org/thanks.php?p=9-7129 m eiheP
http://www.nbptpreservationtrust.org/thanks.php?p=9-8855 DuenCtPhsireoti mWrhn
http://www.nbptpreservationtrust.org/thanks.php?p=9-5683 totrd imrP nno
http://www.nbptpreservationtrust.org/thanks.php?p=9-201 o NpPertvPtonLit hee gemai
http://www.nbptpreservationtrust.org/thanks.php?p=9-8812 thyaln ruePn
http://www.nbptpreservationtrust.org/thanks.php?p=9-7990 nbeEePdaLito eh nmercri
http://www.nbptpreservationtrust.org/thanks.php?p=9-7137 mr savhnePeton
http://www.nbptpreservationtrust.org/thanks.php?p=9-12733 tauneePdrhBnert ry msMaC
http://www.nbptpreservationtrust.org/thanks.php?p=9-5290 eeA ldmPghsniitne
http://www.nbptpreservationtrust.org/thanks.php?p=9-10474 amsro
http://www.nbptpreservationtrust.org/thanks.php?p=9-2860 dTn LonliaOr
http://www.nbptpreservationtrust.org/thanks.php?p=9-6687 dm
http://www.nbptpreservationtrust.org/thanks.php?p=9-6796 inPcee lTniginen Akdn
http://www.nbptpreservationtrust.org/thanks.php?p=9-9083 Qt
http://www.nbptpreservationtrust.org/thanks.php?p=9-357 iaentidtmiid3lp tefm lc oa
http://www.nbptpreservationtrust.org/thanks.php?p=9-10347 nicoaeet lnitosmn
http://www.nbptpreservationtrust.org/thanks.php?p=9-10887 oT
http://www.nbptpreservationtrust.org/thanks.php?p=9-1274 erne
http://www.nbptpreservationtrust.org/thanks.php?p=9-6271 he hNP lmnrmni
http://www.nbptpreservationtrust.org/thanks.php?p=9-7390 F irelth eeneePn
http://www.nbptpreservationtrust.org/thanks.php?p=9-1179 WithetNrsrihieeoPnicot
http://www.nbptpreservationtrust.org/thanks.php?p=9-9250 oanPtie Frs NexRneiSph
http://www.nbptpreservationtrust.org/thanks.php?p=9-2400 nnTIamdAte niDan o oala
http://www.nbptpreservationtrust.org/thanks.php?p=9-11376 pidPoTtemai
http://www.nbptpreservationtrust.org/thanks.php?p=9-10934 Psg1rae0t Ndi Tl0 orc
http://www.nbptpreservationtrust.org/thanks.php?p=9-607 nOieermpnolatSrdgPI intne heip eFrv h
http://www.nbptpreservationtrust.org/thanks.php?p=9-10812 nna omHrthtadrooalaaeinpfImaPm
http://www.nbptpreservationtrust.org/thanks.php?p=9-230 OmrDr ietnOaCreh
http://www.nbptpreservationtrust.org/thanks.php?p=9-7384 BidhPeAe teipryue
http://www.nbptpreservationtrust.org/thanks.php?p=9-1913 dhLOipManeemieeentrinhrP
http://www.nbptpreservationtrust.org/thanks.php?p=9-7733 d l
http://www.nbptpreservationtrust.org/thanks.php?p=9-11962 si
http://www.nbptpreservationtrust.org/thanks.php?p=9-259 vurrtnngrePee Pesn
http://www.nbptpreservationtrust.org/thanks.php?p=9-11511 lar
http://www.nbptpreservationtrust.org/thanks.php?p=9-4040 mrteC P
http://www.nbptpreservationtrust.org/thanks.php?p=9-2558 al
http://www.nbptpreservationtrust.org/thanks.php?p=9-11775 rceaNpss rrttermienhcesnnoi
http://www.nbptpreservationtrust.org/thanks.php?p=9-7645 cd aPdAoroxnrmiia
http://www.nbptpreservationtrust.org/thanks.php?p=9-1275 rooaemamgNerFrh
http://www.nbptpreservationtrust.org/thanks.php?p=9-11846 UaaT
http://www.nbptpreservationtrust.org/thanks.php?p=9-9825 aa
http://www.nbptpreservationtrust.org/thanks.php?p=9-1592 invrPPmOteanehtihe
http://www.nbptpreservationtrust.org/thanks.php?p=9-1895 imheneneit
http://www.nbptpreservationtrust.org/thanks.php?p=9-13253 Vtpnetrlme e ehAsihniSrax

where can i buy valium on the internet
valium online
what are some generic forms of valium
roche valium
cheap valium
valium dosage
where can i buy valium in the uk
mexican valium
what is valium
injecting valium
is valium similar to valerian
valia mpl
valium no presription
can u mix norco with valium
valium for sale
order valium online no prescription
valium high
how can i get valiums without a prescription
order valium online
valium vs xanax
can you snort valium
cat valium
cheap valium online
crushing valium with water to inject
liquid valium
pancakes and a valium
pictures of valium
valia gladcova
valium from india
what type of drug is valium
how to inject valium
ill have a caffe mocha vodka valium latte to go please
robaxin and valium
valium roche
valium withdrawal
when is valium prescribed
xanex or valium which is best/safe
blue valium
buy valium from india or pakistan
can i mix valium and adivan
can xanax or valium help control my anger
can you mix norco with valium
canada valium online
how long is valium in your system
how much is the lethal dose of valium
is it possible to snort valium
liquid valium how to administer
most potent herbal valium substitute
order valium
retrograde amnesia valium
snorting valium
valium & depression
valium administered in endotracheal tube
valium blue
valium dosage for cats
valium expiration
valium for cats
valium western union
caffeine valium
can u mix xanax with valium
can you get high on valium
can you inject valium
can you mix valium with xanax
can you take baking soda with valium
can you take oxycodone and valium at the same time
can you take percocet and valium
can you take robaxin and valium
dosage of valium
generic valium not the same as brand name
how many hours after taking valium can you drink beer
how many hours between valium doses
how strong is 5 milligrams of valium
how to inject liquid valium iv
i need to purchase valium
ibuprofen and valium
kava dosage vs valium
levitra and valium taken together
liquid valium how to administer iv
mri and claustrophobia xanax valium
online valium
oxycodone and valium prescribed together
valium + coma
valium 10
valium and ibuprofen
valium and oxycodone
replica watches
replica rolex watches
replica swiss watches
replica cartier watches
swiss replica watches
replica watches swiss
rado replica watches
swiss made replica watches
best replica watches
coach replica watches
graham replica watches
high end replica watches
replica watches paypal
breitling replica watches
jacob co replica watches
omega replica watches
replica graham watches
replica watches australia
genuine swiss made replica watches
replica gerald genta watches
replica omega watches
rolex replica watches
swiss made breitling replica watches
cheap replica rolex watches
cheap wholesale chinese replica watches
replica lange sohne watches fake
replica olds car watches
replica watches from china
are replica watches legal in china
designer replica watches
james bond replica watches
p_anerai replica watches
repair replica watches
replica bell & ross watches
replica panerai watches
replica watches direct manufacturer
replica watches that except paypal
below wholesale chinese replica watches
brm replica watches
exact replica watches
finest replica watches
ladies replica watches
replica tutima watches
replica ferrari watches
replica hanhardt tutima watches
replica michele watches
replica watches for sale
swiss replica graham watches
swiss-made replica watches
wholesale replica citizen watches
wholesale replica seiko watches
are replica watches legal
best replica rolex watches
perfect replica watches
replica designer watches
replica jacob watches
replica pocket watches
replica porsche watches
replica tutima watches
replica u boat watches
replica watches + 440 stainless steel + automatic movement
watches replica
wholesale italian made replica watches
bell&ross replica watches
best swiss made replica rolex watches
bvlgari replica watches
cartier replica watches
cheap & dirty patek pilippe square dial gold replica watches
cheap replica watches
replica handbags
replica louis vuitton handbags
designer replica handbags
wholesale replica handbags
replica designer handbags
replica coach handbags
replica knockoffs fake handbags
chanel replica handbags
coach replica handbags
replica gucci handbags
designer replica handbags wholesale
hermes replica handbags
wholesale original replica handbags
wholesale replica designer handbags
christian louboutin replica handbags
exact quality replica dooney bourke handbags
replica chanel handbags
wholesale coach replica handbags
louis vuitton replica handbags
replica coach, clearance coach handbags
replica gucci fendi handbags
replica wholesale handbags
cheap replica handbags
importers of replica handbags
replica guicci handbags
replica of gucci handbags
coach replica handbags
gucci replica handbags
guicci replica handbags
louie vuitton replica handbags
replica designer handbags fake look alik
replica handbags and jewelry
replica handbags atlanta
replica handbags wholesale
replica kate spade handbags
replica lv handbags
replica prada handbags
replica prada purses handbags
replica runway handbags
wholesale manufacturer replica handbags
anya hindmarch replica handbags
are coach replica handbags illegal in the us
authentic quality replica handbags
b makowsky replica handbags wholesale
best chanel replica handbags
best hermes replica handbags
bottega veneta replica handbags
burberry replica handbags
designer replica handbags knockoffs
exact replica louis vuitton handbags
high quality replica dooney bourke handbags
kathy van zeeland replica handbags at wholesale prices
knoxville tennessee: replica handbags
louis vuitton replica handbags malaysia
lucky replica handbags
lv handbags replica in malaysia only
mall pushcarts carrying replica handbags
popular replica handbags
replica designer handbags totes
replica handbags uk
replica hermes handbags
replica italian handbags
replica jimmy choo handbags
six star replica handbags
tano handbags knockoff inspired replica
payday loans
no fax payday loans
same day payday loans
direct lender payday loans
payday loans guaranteed no fax
no teletrack payday loans
online payday loans
canada payday loans
faxless payday loans
payday cash loans
payday loans canada
one hour payday loans no faxing
toronto payday loans
no telecheck payday loans
payday loans toronto
best payday loans
direct lender payday loans disability income
court summaries of unethical canadian payday loans
free payday loans
payday loans in canada
direct lenders for payday loans
easy payday loans
fast payday loans
no credit check payday loans
payday loans that don t use teletrack
direct lender no fax payday loans
guaranteed payday loans
payday loans uk
long term payday loans
payday loans online
canadian payday loans
military payday loans
bad credit payday loans
non payday loans
payday loans in 1 hour
payday loans deposited on prepaid debit card
payday loans no bank statement
canadian online payday loans
installment payday loans
one hour payday loans
payday loans for bad credit
payday loans no faxing
personal payday loans
consolidate payday loans
payday loans that dont use teletrack
advance america payday loans
cheap payday loans
cheapest payday loans
faxless same day payday loans
instant payday loans
no faxing payday loans
payday loans cash advances texas
direct payday loans
payday advance loans
1 hour payday loans
90 day payday loans
easy online faxless payday loans
guaranteed approval payday loans
help to consolidate payday loans
i cant pay my payday loans back
payday loans customer login
payday loans from direct lender only
payday loans in new york
payday loans in phoenix area
payday loans without a bank account
30 day payday loans
impact financial payday loans
is there a debt consolidation for payday loans
coach handbags
discount coach handbags
clearance coach handbags
replica coach handbags
fake coach handbags
cheap coach handbags
coach handbags factory for seconds
coach signature knockoff handbags
coach replica handbags
coach handbags in china
coach handbags outlet stores
discounted coach handbags
coach handbags on sale
wholesale coach replica handbags
coach handbags legacy thompson hobotoffee
replacement parts for coach handbags
wholesale cheap coach handbags
coach handbags white
coach handbags wholesale with free shipping
coach legacy handbags
replica coach, clearance coach handbags
where to buy coach handbags in albany ny
authentic coach handbags
coach handbags madison op art ikat wristlet
coach handbags sale
coach inc: is its advantage in luxury handbags sustainable
coach python handbags
wholesale coach handbags
blue coach handbags
buy coach handbags on sale
cheapest coach handbags
coach replica handbags
coach handbags & channel
coach handbags for sale
coach handbags outlet
coach imposter handbags
coach leah handbags
new coach handbags
pink coach hobo handbags
retired coach handbags
westof, coach handbags
wholesale cheap coach handbags usa
are coach replica handbags illegal in the us
authentic coach discount handbags
carly coach handbags
coach discount handbags
coach handbags malaysia
coach handbags strategic buisiness plan edu
coach handbags that are quilted
coach handbags wholesale
coach signature handbags
discount coach ergo style large leather handbags
faux coach handbags
history of coach handbags
imitation coach handbags
knockoff + coach handbags
knockoff coach handbags
navy coach handbags
outlet malls in dallas coach handbags hit bg
real coach handbags
suppliers for coach handbags
airline tickets
cheap airline tickets
cheap airline tickets london
airline tickets italy
cheep airline tickets
dirt cheap airline tickets
low cost airline tickets
robots dirt cheap airline tickets
robots low cost airline tickets
discounted airline tickets
robots cheep airline tickets
robots discounted airline tickets
discount airline tickets
robots cheap airline tickets
robots cheap airline tickets flights
first class low cost airline tickets
cheap air flights low cost airline tickets
airline tickets sexy nude bangkok manila www com
first class dirt cheap airline tickets
aeroflot airline tickets
airline tickets online cheap
first class discounted airline tickets
airline flight tickets cheap airline tickets
first class cheep airline tickets
airline tickets sexy islands hawaii miami florida travel
las vegas hotels airline tickets
cheapest airline tickets
first class cheap airline tickets
dirt cheap airline tickets travl new zealand
first class cheap airline tickets flights
discount airline tickets cheap discounte airline ticket
airline tickets cheep cheep tickets
cheep airline tickets cheep tickets
book airline tickets
cheap cruises carribean airline tickets flights
cheapest plane tickets flights airline flight ticket
airline tickets flights paris france madrid spain
airline tickets paris london rome amsterdam
cheep airline ticket cheep airline tickets
airline tickets discounted flights
cheep airlines cheep airline tickets
first class airline tickets cheap international flights
airline tickets cheap last minute
dirt cheap airline tickets brazil rio de janeiro
airline tickets cheep cheep airline tickets
airline tickets really cheap airline ticket
cheap discounted airline tickets discounted airline tickets
discounted flights airline tickets cheap flights airline
cheep airline tickets com cheep airline tickets
airline tickets cheap
last minute airline tickets
southwest airline tickets
airline tickets student airline fares
cheap airline tickets brazil panama chile
airline tickets for dirt cheap dirt cheap airline tickets
cheap discounted flights airline tickets
cheap discounted airline tickets europe
cheep airline tickets australia cheep airline tickets
dirt cheap airline tickets online airline flight ticket
airline tickets search cheap airfares
airline tickets economy airticket
cheap argentina brazil airline tickets
cheapest airline tickets online cheap air flight ticket
southwest reservation desk airline tickets airlines com
fly cheap airline tickets cheap flights airfares
discount airline tickets argentina brazil
discounted airline tickets discounted airline tickets
cheep airline tickets london cheep airline tickets
airline tickets cheap international flights
airline tickets northwest airline
last minute airline tickets online cheap air flights
business economy airline tickets flights
low cost airline tickets low cost airline tickets
airline tickets delta airline
airline cheap dirt ticket dirt cheap airline tickets
airline tickets dirt cheap dirt cheap airline tickets
cheap airlines tickets
manila cebu airline tickets hotels
domestic airline tickets domestic travel
discount airline tickets airline discount central
airline tickets discount airline ticket
low cost cheap airfares low cost airline tickets
dirt cheap airline ticket dirt cheap airline tickets
dirt cheap airline tickets usa cheap flights from usa
dirt cheap airline tickets dirt cheap airline tickets
airline tickets discounts flights cheap flights airfares
dirt cheap airline tickets online cheap flights airfares
cheap international airline tickets
airline tickets cheap airline fares
discount military airline tickets airline discount central
cheap low cost airline tickets low cost airline tickets
low cost tickets south america low cost airline tickets
low cost airline ticket low cost airline tickets
airline tickets cheap first class airfares
coach class airfares low cost airline tickets
low cost economy ticket low cost airline tickets
low cost car rentals low cost airline tickets
australia sydney hotels flights airline tickets
australia airline tickets travel
continental airline tickets

Filed Under (Random) by mark on 07-01-2007

Figured I might as well use this piece of software since I have to maintain it.

Basically to summarise my experiences with SATA RAID controllers (see here for the full details) under a basic stress load:

1. Adaptec 21610SA. Junk, very slow, locks up after about 4 hours.
2. 3ware 9550. Kicks drives out after about 10 minutes.

Then I got an Areca 1160. This works, kind of. It corrupts data, this is directly correlated with a ‘media error count’ that is visible only on the individual, detailed status page for each drive available through the card’s HTTP interface. After getting in touch with Areca they advised me that this was either power (I tried it with a pair of PSUs with the load split roughly evenly, so I don’t think that this is really the case), or it’s an issue with the firmware of the drives (16xWD320RE SATA disks).

However, entirely turning off the write cache makes the thing work OK, and it has done this for about a year. The write speed is absolutely pitiful though (and writes also kill reads). I get ~15Mb/s in moderately seek bound workloads, and ~30Mb/s for sequential writes. Hurrah, slower than some of my really old single IDE disks! Oh well, it’s good enough for a backup/TV server at any rate. Overall SATA RAID is still totally pants though!
http://www.karen-rollins.com/Trecena/?p=10413 DeaCofiph
http://www.karen-rollins.com/Trecena/?p=12646 hiclPKru ytr
http://www.karen-rollins.com/Trecena/?p=9629 peguCh
http://www.karen-rollins.com/Trecena/?p=76 ilx
http://www.karen-rollins.com/Trecena/?p=2825 nre
http://www.karen-rollins.com/Trecena/?p=256 puRq
http://www.karen-rollins.com/Trecena/?p=242 d
http://www.karen-rollins.com/Trecena/?p=336 I lenlc
http://www.karen-rollins.com/Trecena/?p=280 Sot
http://www.karen-rollins.com/Trecena/?p=8145 aydatuc
http://www.karen-rollins.com/Trecena/?p=14557 unr
http://www.karen-rollins.com/Trecena/?p=18869 e
http://www.karen-rollins.com/Trecena/?p=12793 no Paxhieusca
http://www.karen-rollins.com/Trecena/?p=17959 arCy
http://www.karen-rollins.com/Trecena/?p=4667 ltBuya
http://www.karen-rollins.com/Trecena/?p=10938 uy
http://www.karen-rollins.com/Trecena/?p=15838 nliBuriPy
http://www.karen-rollins.com/Trecena/?p=17791 rapce
http://www.karen-rollins.com/Trecena/?p=16475 la
http://www.karen-rollins.com/Trecena/?p=8096 ABnahuy
http://www.karen-rollins.com/Trecena/?p=19023 eB
http://www.karen-rollins.com/Trecena/?p=17119 vr SrtnrOd
http://www.karen-rollins.com/Trecena/?p=16041 P
http://www.karen-rollins.com/Trecena/?p=14221 eM ok
http://www.karen-rollins.com/Trecena/?p=157 vLi hHe
http://www.karen-rollins.com/Trecena/?p=414 aopmhrT
http://www.karen-rollins.com/Trecena/?p=9167 ruacP aCd
http://www.karen-rollins.com/Trecena/?p=359 niiL
http://www.karen-rollins.com/Trecena/?p=135 oniv
http://www.karen-rollins.com/Trecena/?p=18771 usXrahPel
http://www.karen-rollins.com/Trecena/?p=187 ieLosl
http://www.karen-rollins.com/Trecena/?p=12275 aIho smep
http://www.karen-rollins.com/Trecena/?p=6787 CZape hor
http://www.karen-rollins.com/Trecena/?p=223 etOrc
http://www.karen-rollins.com/Trecena/?p=143 aEvts
http://www.karen-rollins.com/Trecena/?p=142 aveecr
http://www.karen-rollins.com/Trecena/?p=1531 ayl
http://www.karen-rollins.com/Trecena/?p=416 ohm
http://www.karen-rollins.com/Trecena/?p=15299 lu
http://www.karen-rollins.com/Trecena/?p=228 e
http://www.karen-rollins.com/Trecena/?p=200 esnnaM
http://www.karen-rollins.com/Trecena/?p=227 Pila
http://www.karen-rollins.com/Trecena/?p=8936 reOdALC
http://www.karen-rollins.com/Trecena/?p=11862 clHnu
http://www.karen-rollins.com/Trecena/?p=6675 iy Zti
http://www.karen-rollins.com/Trecena/?p=14620 r
http://www.karen-rollins.com/Trecena/?p=7900 vya raBAu
http://www.karen-rollins.com/Trecena/?p=4894 reulxaCsechPbeer
http://www.karen-rollins.com/Trecena/?p=7858 dubteaO
http://www.karen-rollins.com/Trecena/?p=128 o
http://www.karen-rollins.com/Trecena/?p=160 pc
http://www.karen-rollins.com/Trecena/?p=13206 esCripe
http://www.karen-rollins.com/Trecena/?p=354 d
http://www.karen-rollins.com/Trecena/?p=10119 naaC
http://www.karen-rollins.com/Trecena/?p=18400 G-CVhal pe
http://www.karen-rollins.com/Trecena/?p=355 GmrtaounhwH
http://www.karen-rollins.com/Trecena/?p=10203 u
http://www.karen-rollins.com/Trecena/?p=4769 rexeC Oald
http://www.karen-rollins.com/Trecena/?p=8838 ierrrd OB
http://www.karen-rollins.com/Trecena/?p=7907 dvaraerOrA
http://www.karen-rollins.com/Trecena/?p=18043 moxy
http://www.karen-rollins.com/Trecena/?p=17259 yBhuSall ia
http://www.karen-rollins.com/Trecena/?p=79 aaAvr
http://www.karen-rollins.com/Trecena/?p=7473 haAaC p
http://www.karen-rollins.com/Trecena/?p=13626 Lxein auhe
http://www.karen-rollins.com/Trecena/?p=14627 ppCheiot
http://www.karen-rollins.com/Trecena/?p=425 lm Vpa
http://www.karen-rollins.com/Trecena/?p=11428 tnFoluB ye
http://www.karen-rollins.com/Trecena/?p=2806 yn
http://www.karen-rollins.com/Trecena/?p=12163 rx teimI
http://www.karen-rollins.com/Trecena/?p=4861 bC e
http://www.karen-rollins.com/Trecena/?p=12849 uxiLBsa
http://www.karen-rollins.com/Trecena/?p=5 vi
http://www.karen-rollins.com/Trecena/?p=362 hebraeiP
http://www.karen-rollins.com/Trecena/?p=15901 oPCepada
http://www.karen-rollins.com/Trecena/?p=16272 c aozursrP
http://www.karen-rollins.com/Trecena/?p=175 tlK
http://www.karen-rollins.com/Trecena/?p=217 rhosN
http://www.karen-rollins.com/Trecena/?p=7550 AdPhcetaun
http://www.karen-rollins.com/Trecena/?p=16874 edctRo
http://www.karen-rollins.com/Trecena/?p=9426 eOriiad
http://www.karen-rollins.com/Trecena/?p=17945 ihSdB
http://www.karen-rollins.com/Trecena/?p=17896 Biua sutS
http://www.karen-rollins.com/Trecena/?p=5192 f lCcuhDp
http://www.karen-rollins.com/Trecena/?p=5290 hpeirEedah penC
http://www.karen-rollins.com/Trecena/?p=7228 eh
http://www.karen-rollins.com/Trecena/?p=155 ofer
http://www.karen-rollins.com/Trecena/?p=7270 d rcrhpiexeA
http://www.karen-rollins.com/Trecena/?p=288 iTcror
http://www.karen-rollins.com/Trecena/?p=6717 asruc
http://www.karen-rollins.com/Trecena/?p=7081 aa
http://www.karen-rollins.com/Trecena/?p=9468 dBilmuy Co
http://www.karen-rollins.com/Trecena/?p=17168 or
http://www.karen-rollins.com/Trecena/?p=49 aUl
http://www.karen-rollins.com/Trecena/?p=300 Xeld
http://www.karen-rollins.com/Trecena/?p=9573 Oodi
http://www.karen-rollins.com/Trecena/?p=12303 aIuosPhesmrc
http://www.karen-rollins.com/Trecena/?p=9951 y orParao
http://www.karen-rollins.com/Trecena/?p=15110 arOPdrreod el
http://www.karen-rollins.com/Trecena/?p=10301 yDou
http://www.karen-rollins.com/Trecena/?p=9174 yCoBsdxae
http://www.karen-rollins.com/Trecena/?p=15705 maiC
http://www.karen-rollins.com/Trecena/?p=15397 y
http://www.karen-rollins.com/Trecena/?p=12709 siOm
http://www.karen-rollins.com/Trecena/?p=12205 xr
http://www.karen-rollins.com/Trecena/?p=5522 nHcddyoerrreoo
http://www.karen-rollins.com/Trecena/?p=8880 Bur
http://www.karen-rollins.com/Trecena/?p=7417 edotr
http://www.karen-rollins.com/Trecena/?p=15159 xdr Oi
http://www.karen-rollins.com/Trecena/?p=8390 dyBiAa
http://www.karen-rollins.com/Trecena/?p=10693 esy
http://www.karen-rollins.com/Trecena/?p=18526 dra
http://www.karen-rollins.com/Trecena/?p=3 lzrlApmoa
http://www.karen-rollins.com/Trecena/?p=2 epdi
http://www.karen-rollins.com/Trecena/?p=230 ei
http://www.karen-rollins.com/Trecena/?p=8110 dwganeaACha
http://www.karen-rollins.com/Trecena/?p=2701 eeshnaucTt
http://www.karen-rollins.com/Trecena/?p=1696 aaCnn iauBd
http://www.karen-rollins.com/Trecena/?p=4471 tayvB unA
http://www.karen-rollins.com/Trecena/?p=14508 shpPr searunoy
http://www.karen-rollins.com/Trecena/?p=97 hriB
http://www.karen-rollins.com/Trecena/?p=5751 ea tO
http://www.karen-rollins.com/Trecena/?p=6780 r
http://www.karen-rollins.com/Trecena/?p=11470 otvh
http://www.karen-rollins.com/Trecena/?p=5410 iraeio tFhrccPu
http://www.karen-rollins.com/Trecena/?p=262 Rcrltlao
http://www.karen-rollins.com/Trecena/?p=7312 oABu acp
http://www.karen-rollins.com/Trecena/?p=15208 n
http://www.karen-rollins.com/Trecena/?p=9237 ax
http://www.karen-rollins.com/Trecena/?p=47 iVidocn
http://www.karen-rollins.com/Trecena/?p=233 i
http://www.karen-rollins.com/Trecena/?p=18204 TC
http://www.karen-rollins.com/Trecena/?p=8306 pgnet
http://www.karen-rollins.com/Trecena/?p=179 rm
http://www.karen-rollins.com/Trecena/?p=7158 lpshPcca
http://www.karen-rollins.com/Trecena/?p=119 Crsre
http://www.karen-rollins.com/Trecena/?p=16027 moPrecuthu ieraPs
http://www.karen-rollins.com/Trecena/?p=101 la
http://www.karen-rollins.com/Trecena/?p=13136 urteihsLveha
http://www.karen-rollins.com/Trecena/?p=18190 muTo
http://www.karen-rollins.com/Trecena/?p=13976 uoracBM evy
http://www.karen-rollins.com/Trecena/?p=8187 h
http://www.karen-rollins.com/Trecena/?p=358 viun
http://www.karen-rollins.com/Trecena/?p=282 nrpuaSem
http://www.karen-rollins.com/Trecena/?p=7319 rimdApelro
http://www.karen-rollins.com/Trecena/?p=14655 oPsNeoihpcatmr
http://www.karen-rollins.com/Trecena/?p=9475 milC
http://www.karen-rollins.com/Trecena/?p=15887 od
http://www.karen-rollins.com/Trecena/?p=276 S
http://www.karen-rollins.com/Trecena/?p=13199 Orlere
http://www.karen-rollins.com/Trecena/?p=14172 i BcouMy
http://www.karen-rollins.com/Trecena/?p=12009 ieHuhoaPoc r
http://www.karen-rollins.com/Trecena/?p=367 xxe
http://www.karen-rollins.com/Trecena/?p=17651 i
http://www.karen-rollins.com/Trecena/?p=19030 eOZre
http://www.karen-rollins.com/Trecena/?p=11337 xleF
http://www.karen-rollins.com/Trecena/?p=11372 oacrPuhxamlF
http://www.karen-rollins.com/Trecena/?p=12695 seuPrcahalm Lct
http://www.karen-rollins.com/Trecena/?p=392 Liutyv aB
http://www.karen-rollins.com/Trecena/?p=15943 doer
http://www.karen-rollins.com/Trecena/?p=91 rAat
http://www.karen-rollins.com/Trecena/?p=10364 iepcnhDa abCe
http://www.karen-rollins.com/Trecena/?p=224 loty r
http://www.karen-rollins.com/Trecena/?p=18792 eXCeor
http://www.karen-rollins.com/Trecena/?p=15635 as
http://www.karen-rollins.com/Trecena/?p=17749 ypBu
http://www.karen-rollins.com/Trecena/?p=6577 Zs B
http://www.karen-rollins.com/Trecena/?p=90 v
http://www.karen-rollins.com/Trecena/?p=14270 tByMnor
http://www.karen-rollins.com/Trecena/?p=8495 td Aadrrv
http://www.karen-rollins.com/Trecena/?p=426 h beCax
http://www.karen-rollins.com/Trecena/?p=10098 eceuo thrs
http://www.karen-rollins.com/Trecena/?p=5989 cuoaB
http://www.karen-rollins.com/Trecena/?p=313 A
http://www.karen-rollins.com/Trecena/?p=16958 c gireo
http://www.karen-rollins.com/Trecena/?p=3496 DzecuPa aiarpem
http://www.karen-rollins.com/Trecena/?p=9370 iuarynleBCx
http://www.karen-rollins.com/Trecena/?p=434 dar eTh
http://www.karen-rollins.com/Trecena/?p=16664 e
http://www.karen-rollins.com/Trecena/?p=14480 ppnhNyra eaoCs
http://www.karen-rollins.com/Trecena/?p=58 aZn
http://www.karen-rollins.com/Trecena/?p=226 are
http://www.karen-rollins.com/Trecena/?p=12842 msaa
http://www.karen-rollins.com/Trecena/?p=198 rLoynal
http://www.karen-rollins.com/Trecena/?p=5940 B
http://www.karen-rollins.com/Trecena/?p=10945 O
http://www.karen-rollins.com/Trecena/?p=8012 hpr mieC
http://www.karen-rollins.com/Trecena/?p=89 a
http://www.karen-rollins.com/Trecena/?p=302 Z
http://www.karen-rollins.com/Trecena/?p=337 lofsanolP
http://www.karen-rollins.com/Trecena/?p=7662 rOctee dalrA
http://www.karen-rollins.com/Trecena/?p=11477 sB yomxa
http://www.karen-rollins.com/Trecena/?p=21 xEo
http://www.karen-rollins.com/Trecena/?p=11785 cdHiehm Ccia
http://www.karen-rollins.com/Trecena/?p=7130 iarepucA
http://www.karen-rollins.com/Trecena/?p=5508 e
http://www.karen-rollins.com/Trecena/?p=15544 yanrB Pdn
http://www.karen-rollins.com/Trecena/?p=12751 aLu
http://www.karen-rollins.com/Trecena/?p=13486 s
http://www.karen-rollins.com/Trecena/?p=10070 Ccpttohaey eC
http://www.karen-rollins.com/Trecena/?p=10980 rEsrcuahuP x
http://www.karen-rollins.com/Trecena/?p=17413 inlarrugdi reOS
http://www.karen-rollins.com/Trecena/?p=5800 rie
http://www.karen-rollins.com/Trecena/?p=7361 tni
http://www.karen-rollins.com/Trecena/?p=17602 r iBStlu
http://www.karen-rollins.com/Trecena/?p=7648 glPAreeausal
http://www.karen-rollins.com/Trecena/?p=150 lan
http://www.karen-rollins.com/Trecena/?p=17462 Ond roofreSfr
http://www.karen-rollins.com/Trecena/?p=16566 ahtrRssu
http://www.karen-rollins.com/Trecena/?p=11050 EiaCvha ts
http://www.karen-rollins.com/Trecena/?p=12555 Bl
http://www.karen-rollins.com/Trecena/?p=5178 ni
http://www.karen-rollins.com/Trecena/?p=10056 ut toBeCyc
http://www.karen-rollins.com/Trecena/?p=8985 nCad
http://www.karen-rollins.com/Trecena/?p=9132 eO
http://www.karen-rollins.com/Trecena/?p=144 Elexno
http://www.karen-rollins.com/Trecena/?p=7263 Bu eAxpihcy
http://www.karen-rollins.com/Trecena/?p=11932 Haep
http://www.karen-rollins.com/Trecena/?p=16979 hplCea aumaaR
http://www.karen-rollins.com/Trecena/?p=8348 AnervOe mddaatr
http://www.karen-rollins.com/Trecena/?p=17301 Srlhuiakc
http://www.karen-rollins.com/Trecena/?p=18099 erd daOtranTr
http://www.karen-rollins.com/Trecena/?p=290 liaTrp
http://www.karen-rollins.com/Trecena/?p=14823 eo
http://www.karen-rollins.com/Trecena/?p=415 cheCoiietp F
http://www.karen-rollins.com/Trecena/?p=13843 s
http://www.karen-rollins.com/Trecena/?p=10546 yfi urifn
http://www.karen-rollins.com/Trecena/?p=18533 uotaVcs
http://www.karen-rollins.com/Trecena/?p=7116 BcrAyiup
http://www.karen-rollins.com/Trecena/?p=11624 Bytoeu efrG
http://www.karen-rollins.com/Trecena/?p=13584 ateux oLnBi
http://www.karen-rollins.com/Trecena/?p=13430 eroLsrP
http://www.karen-rollins.com/Trecena/?p=10259 Dderrlyers Oe
http://www.karen-rollins.com/Trecena/?p=15019 payh
http://www.karen-rollins.com/Trecena/?p=7620 a pClaleA
http://www.karen-rollins.com/Trecena/?p=12597 clhreaPuf Ksote
http://www.karen-rollins.com/Trecena/?p=13283 ralisrop chu
http://www.karen-rollins.com/Trecena/?p=17497 fsaeou Pnorr
http://www.karen-rollins.com/Trecena/?p=16328 BP et
http://www.karen-rollins.com/Trecena/?p=9314 ssiu
http://www.karen-rollins.com/Trecena/?p=8250 trvA d
http://www.karen-rollins.com/Trecena/?p=12716 iasah li
http://www.karen-rollins.com/Trecena/?p=9755 PCg
http://www.karen-rollins.com/Trecena/?p=260 bontainmR
http://www.karen-rollins.com/Trecena/?p=13451 x
http://www.karen-rollins.com/Trecena/?p=16174 rP
http://www.karen-rollins.com/Trecena/?p=11148 mhFpara
http://www.karen-rollins.com/Trecena/?p=3151 rrUed
http://www.karen-rollins.com/Trecena/?p=10399 fi ouk
http://www.karen-rollins.com/Trecena/?p=18232 mrixcePT uh
http://www.karen-rollins.com/Trecena/?p=159 cmolnHi
http://www.karen-rollins.com/Trecena/?p=1454 ud yreBi
http://www.karen-rollins.com/Trecena/?p=124 nDaalzo
http://www.karen-rollins.com/Trecena/?p=365 iFaa e
http://www.karen-rollins.com/Trecena/?p=6472 euhtrcPmls
http://www.karen-rollins.com/Trecena/?p=105 Cxaeso
http://www.karen-rollins.com/Trecena/?p=13983 Oovdc ra
http://www.karen-rollins.com/Trecena/?p=4957 pdzCre anoreOlm
http://www.karen-rollins.com/Trecena/?p=18337 ai Br
http://www.karen-rollins.com/Trecena/?p=6433 enuhxXPacarsa
http://www.karen-rollins.com/Trecena/?p=10840 s mEmyaB
http://www.karen-rollins.com/Trecena/?p=210 ybutolMm
http://www.karen-rollins.com/Trecena/?p=15642 Ps
http://www.karen-rollins.com/Trecena/?p=9566 CfdnyoiBou
http://www.karen-rollins.com/Trecena/?p=296 sVtcea
http://www.karen-rollins.com/Trecena/?p=7795 rPlhmAoesxuic
http://www.karen-rollins.com/Trecena/?p=127 eyl
http://www.karen-rollins.com/Trecena/?p=24 iceiot
http://www.karen-rollins.com/Trecena/?p=275 oaSrr
http://www.karen-rollins.com/Trecena/?p=11323 rhrmecuP Fsc
http://www.karen-rollins.com/Trecena/?p=10147 o
http://www.karen-rollins.com/Trecena/?p=16223 oeti
http://www.karen-rollins.com/Trecena/?p=5849 dOrMhreacer
http://www.karen-rollins.com/Trecena/?p=8236 ceuaatPrsx
http://www.karen-rollins.com/Trecena/?p=17644 uxarrP cs
http://www.karen-rollins.com/Trecena/?p=11820 uB iinlcm
http://www.karen-rollins.com/Trecena/?p=312 rt
http://www.karen-rollins.com/Trecena/?p=7123 erldu
http://www.karen-rollins.com/Trecena/?p=13647 ezClLpoao
http://www.karen-rollins.com/Trecena/?p=15495 Playue B
http://www.karen-rollins.com/Trecena/?p=6361 odic
http://www.karen-rollins.com/Trecena/?p=9818 rd oarzra
http://www.karen-rollins.com/Trecena/?p=167 radlneI
http://www.karen-rollins.com/Trecena/?p=5529 daon dyhe
http://www.karen-rollins.com/Trecena/?p=5905 eauxh eiNCpm
http://www.karen-rollins.com/Trecena/?p=9664 rnouCedo r
http://www.karen-rollins.com/Trecena/?p=4776 leC Ce
http://www.karen-rollins.com/Trecena/?p=12415 adgamra
http://www.karen-rollins.com/Trecena/?p=18638 Vermd
http://www.karen-rollins.com/Trecena/?p=715 irtrnhne
http://www.karen-rollins.com/Trecena/?p=332 eSbpleee( Wal
http://www.karen-rollins.com/Trecena/?p=12471 a
http://www.karen-rollins.com/Trecena/?p=13143 icon
http://www.karen-rollins.com/Trecena/?p=10784 lsculr
http://www.karen-rollins.com/Trecena/?p=16685 a
http://www.karen-rollins.com/Trecena/?p=234 a ellt
http://www.karen-rollins.com/Trecena/?p=15558 nnhaaeCiP
http://www.karen-rollins.com/Trecena/?p=4562 nuiaPx
http://www.karen-rollins.com/Trecena/?p=12611 rOrl td

Filed Under (Random) by mark on 14-04-2006

For the last 4 months, I’ve had a 4TB storage array (=16x320Gb WD RE SATA hard disks) sitting in my front room.

Doing nothing. Nothing at all.

The reason? SATA RAID cards are _unfeasibly_ bad. Here’s what I’ve tried so far, under linux (various kernel versions and distributions):

- an Adaptec 21610SA. This lasts about 4 hours before it does the classic aacraid: scsi hang? (google for it, lots of references). Adaptec sent me a “heat sync” (sic.) for some reason, in a russian doll style series of packaging.
- a 3ware 9550. This lasts about 10 minutes before it powers down a random drive (physically: you can hear it), then claims that the drive has been reset (well, duh). 3ware deny everything (the last thing they told me was that it was “noise on the PCI bus”. The 3ware card has been tried in a couple of different motherboards, same thing in both of them).

Is this isolated? From friends I know of:

- At least two other machines containing Adaptec AAC series cards with the scsi hang? issue;
- At least two other machines containing 3ware 9xxx cards that pop drives;
- At least one other machine containing a 3ware 9xxx card that pops drives under a different OS

Good news? I got to play with an Areca-1160. Managed 24 hours without an issue, so that’s the best so far. If I can find someone who distributes them in the UK, I’ll get one!

Random technical details:

- I break the cards with dd if=/dev/sdX of=/dev/null (x10); dd if=/dev/zero of=fooX (x10); bonnie++ (x10); and cp -a on a kernel tree (x10). IoW, just a bit of IO.
- The cards were tried in a P8SCi motherboard, 3Ghz xeon, plenty of power supply (including 1100W worth for some experiments)
- The P8SCi is on 3ware’s compatibility list
- 3ware card also tried in an Athlon64+H8SSL motherboard setup with a 760W power supply

I can’t believe these companies get away with peddling such crap. But then I suppose so do Highpoint and Promise to name a few I’ve seen shite (ATA-100/ATA-66) hardware off in the past.

Is SCSI any better?!

Filed Under (Random) by mark on 14-04-2006

Updated to latest version (0.88.1). I’ve updated in the interm; will try to post other updates in a timely fashion. Do let me know if you are using this, that will be an incentive to do so.

Other fixes are:

- Updated to build with VC8 (VS2005)
- Fix a number of extraneous close() calls that VC8 doesn’t like
- Fix service stub so that it works with w2k3

You can get the MSI installer & patch here.

Full source code can be downloaded here.

Filed Under (Random) by mark on 15-01-2006


Latest: added a version that compiles with 2.6.27 kernels.
Latest: if you are packaging this, see the note at the bottom as to why I don’t provide a simple patch.
Latest: added an updated version that compiles with 2.6.24 kernels, tested against
Latest: updated `firmware` to a later revision from hauppauge (743 codesets)


  1. This is not for the MCE version of the PVR-150. This includes a USB unit which does not work the same way as the normal PVR-150, use standard lirc (and lirc_mceusb) for that.
  2. Use a recent kernel (note: the ivtv drivers have been part of the mainline kernel for a good number of revisions so no external drivers are required — sorry, I don’t know off hand which revision they were merged in), or for older kernels install ivtv-0.4.2+ (from this page). Earlier versions of ivtv _are not supported_.
  3. Get the pre-patched lirc 0.8.5-CVS-pvr150 tarball. There are also earlier versions that can be found here, should you want them. The previous revision may be required for kernels < 2.6.27 (as it is untested on lower revisions -- in theory it should work)
  4. You need the dialog package installed to use the lirc configuration GUI, so install that (apt-get install dialog, yum install dialog, whatever is appropriate for your distribution).
  5. Unpack the patched lirc:

    cd /usr/src
    tar xfj lirc-0.8.3-CVS-pvr150-2.tar.bz2
    cd lirc-0.8.3-CVS-pvr150-2

    TV card
    i – Hauppauge PVR-150 TV card (note: _NOT_ ‘g – Hauppauge TV card’)
    Save configuration & run configure


    make && make install.

  6. IR blaster only: Now you need the ‘firmware’. This is a set of data blocks that correspond to those generated by the windows software. This goes in /usr/lib/hotplug/firmware on my debian system. Depending on your system this might also be /usr/local/lib/firmware, /lib/firmware or /lib/modules.

    Note that the entire firmware is kept in memory (currently 300K) so this makes the driver quite large. (I have no plans to sort this out, memory is cheap).

  7. Check everything is working so far:

    modprobe lirc_dev debug=1 && modprobe lirc_pvr150 debug=1

    Check the syslog output. This should report something like:

    Aug 28 02:09:11 soapbox kernel: lirc_pvr150: chip found with RX and TX
    Aug 28 02:09:11 soapbox kernel: ivtv: i2c attach [client=Hauppauge PVR150 IR RX, ok]
    Aug 28 02:09:11 soapbox kernel: ivtv: i2c attach [client=Hauppauge PVR150 IR TX, ok]
    Aug 28 02:09:11 soapbox kernel: lirc_dev: lirc_register_plugin: sample_rate: 0
    Aug 28 02:09:11 soapbox udev[5221]: creating device node ‘/dev/lirc0′
    Aug 28 02:09:11 soapbox kernel: lirc_pvr150: firmware of size 302355 loaded
    Aug 28 02:09:11 soapbox kernel: lirc_pvr150: 743 codesets loaded
    Aug 28 02:09:11 soapbox kernel: lirc_pvr150: Hauppauge PVR-150 IR blaster: firmware version 1.3.0

    This means that the driver has detected and initialised the IR blaster hardware — if you don’t see that then let me know.

  8. You need to configure lircd, and find out which codeset you are going to be using. The easiest way is to start with this configuration file which contains key definitions for everything in the database. Do not use other lirc configuration files for specific STBs — these simply will not work. The IR chip is only capable of sending those codes which are in the database.
  9. Start lircd. Note: if you are using a static /dev, you may need to make a device for lirc. If unsure, once you have verified that the module has been loaded ok, run ls -l /dev/lirc*. If you don’t see a /dev/lirc0 or similar, then try mknod /dev/lirc0 c 61 0 if the steps below fail.

    modprobe lirc_dev && modprobe lirc_pvr150 debug=1
    lircd --device=/dev/lirc0

  10. You can now check if the remote is working using irw. Run this, and press buttons on the remote. You should see some output like

    0000000000001795 00 Down Hauppauge_350.

  11. Next, for the ir blaster you need to work out which codeset to use, this is the tricky bit. For this I have send_power_new, a script that just sends the power command in every single codeset. You may find your codeset number listed here if you are lucky.

    Firstly, check that you are seeing the IR blaster blink. If you don’t have blinking lights at this stage, your cable probably isn’t in the card properly (try wiggling it around), or it may be broken.

    Next you need to stick the IR blaster on the IR receiver of box that you intend to control, being quite careful to position it correctly — it has a very short range (a few cm) and took me a couple of goes to get right. The best way to do this is to find the IR demodulator on the box — easiest with a torch. Note that this is _not_ the light that comes on when you press a button on the remote, they tend to look like this.

    If you can’t get this to work, please try and check it against the Windows driver if possible. If your device will work with the windows driver but not my driver, then it’s a driver bug that I should be able to fix. If it does not work with the Windows driver either then your only options are to use a different IR blaster or bug hauppauge until they add support for the box you are trying to control. At that point, I can update my
    database too.

  12. Once you know which codeset you want you can go and delete all of the rest from lircd.conf. They are named “XXX_key” so should be pretty easy to find. I also gave the keys standard names (0-9).
  13. To get mythtv to work, configure a channel change script for your device. There’s one here that should work out of the box if you
    rename the number keys.
  14. If you’re happy, you can always send me beer money. If not, add comments at the bottom :)

That’s it, good luck!


The lirc distribution tarball is generated using `make dist-bzip2` which uses the gnu autotools to generate a configure script, and Makefile.in, etc. The contents of these generated files depends very much on the version of autotools that is installed; this varies from distribution to distribution. I don’t have the same auto* as the lirc maintainers, so producing a patch file against a distribution tarball makes a patch bigger than the original source archive. Hence I don’t bother; I just make a new dist bzip2 and drop it here.

I maintain the code by importing a current lirc CVS into a subversion repository hosted on this server. I tag each lirc import, generate a diff from the previous lirc import and apply it to my source tree, then port any fixes from lirc_i2c.c to lirc_pvr150.c, test, and generate a new .tar.bz2. To get the source tree you can do:

svn co http://svn.blushingpenguin.com/svn/trunk/3rdparty/lirc lirc

and you can get the changes I made to the CVS revision of lirc with:

svn diff http://svn.blushingpenguin.com/svn/vendor/lirc/current http://svn.blushingpenguin.com/svn/trunk/3rdparty/lirc

Filed Under (Random) by mark on 04-10-2005

This patch, with some fixes, has gone into the 0.4.2 release of ivtv. Please use ivtv 0.4.2+ rather than following the instructions here. You can get this from the ivtv site.

This is related to my IR blaster driver, in case that’s what you are looking for. Do read on though, it might be informative! If you are just looking for patch instructions, dive right to the end to skip all this detail.

A while ago I looked around to see if the hauppauge windows driver for the remote control had any issues. I found quite a few people who claimed that it stopped responding, and that they had to reset the hauppauge software, etc and one or two pointers to a new (beta) driver on the hauppauge site. This was fairly intriguing as among the release notes it lists “i2c (IR blaster) fixes”. I managed to find time this weekend to take the driver apart, and see exactly what they did. Basically they rewrote the i2c support.

Some background, in case you aren’t familar: i2c is a low speed (varies, say ~100-400KHz) 2 wire serial bus for interconnecting various random chips. Because it’s so useful, it actually gets used all over the place to ‘glue’ various chips together on a board and then let the OS driver control them. For example, on your typical PVR-150 card you’ll find at least tuner, video encoder, IR chip and eeprom (configuration data). These are all i2c connected, and the driver talks i2c to them to program them (as well as other memory mapped registers, but let’s stick to the issue).

In some cases, the i2c protocol is implemented by special hardware on one of the chips (e.g. the bt878a chip has an i2c controller in it). This is the nice case, since the CPU doesn’t have to do much in the way of work. In the cheap case, you get to control the clock & data lines yourself from the OS (a kind of winmodem approach!). The hauppauge stuff takes the latter approach — indeed so does a bunch of other hardware. It’s so common that there is a module in the linux i2c subsystem specifically for doing this — it’s called i2c-algo-bit, and the approach is known as ‘bit-banging’. As a side issue, bit-banging takes up some constant fraction of CPU, because it has to use busy delays (it’s not possible to get microsecond accurate sleeps in the kernel, so we must run the CPU in a loop). Since these loops are constant _time_ you’ll find that whatever the CPU the IR polling (for example) takes a constant amount of time. For lirc_i2c, this works out at about 1%, and for my lirc_pvr150 module (that sends a ‘poll’ command to the receiver as well to see if it is borked or not), it’s about 2%. This is why I think the hardware approach is better, but whatever, we are stuck with it.

From the looks of the hauppauge driver changes it appears that the classical bit banging algorithm causes problems with the IR chip that the PVR-150 uses. I disassembled both the new and old drivers, and the older driver uses something very similar to the approach of i2c-algo-bit — it uses KeStallExecutionProcessor, or a busy loop, to implement delays. The newer driver dispenses with this, and instead polls the lines to see when they have become high/low and uses memory mapped register reads for stalling. I’m guessing that those reads are (roughly) constant time because they have to go across the PCI bus. I implemented the newer method, and low and behold this seems to solve my problems. I’ve not seen an IR chip reset by my lirc_pvr150 driver in 16 hours now (update: now 28), whereas I was getting them _much_ more frequently beforehand (dozens a day). This stops the remote dropping out if the chip has to reset, which it did do a few times a day for seconds at a time. So I’m happier. Anyway, if anyone else would like to try it, here’s the HOWTO part…

The HOWTO part

This patch is against the latest SVN version of ivtv. Things are changing rapidly with ivtv, and there is currently no really stable release (as you’ve probably realised). It might apply against the 0.3.8 release, if you try that and it fails let me know and I’ll post a patch against that version. It should go into the driver, eventually, if it really does sort out the IR chip issues.

Get the ivtv code from svn (you will need to apt-get install svn or whatever first!). See this page for detailed instructions.

svn co http://ivtvdriver.org/svn/ivtv/trunk ivtv

Download the i2c patch from the trac ticket for this issue (I will keep this up to date), or from from here. Apply the patch:

cd ivtv/driver
patch -p0 < i2c.patch
make && make install
cd ../utils
make && make install

Then reboot. Please _do not_ just rmmod ivtv & insmod ivtv — this won’t work. If you don’t want to reboot, rmmod _all_ of the ivtv modules first, especially cx25840. If you don’t do that, you’ll get an oops (then reboot, no biggie).
Note that this patch also covers section #3 of the IR blaster HOWTO, so you don’t need to try and apply both patches (if you do that, it will certainly fail to apply cleanly).

That’s it. You should stop seeing log messages related to resetting the IR chip, and the remote shouldn’t drop out at all. That’s the idea anyway. Any feedback would be appreciated!



Filed Under (Random) by mark on 04-10-2005

I bought one of these a while ago, and discovered that the remote control didn’t work. After lots of looking at the board, and the help of an accomplished friend with a multimeter we were able to discover roughly what it does. In essence, it’s really dumb hardware :) There’s an IR demodulator on the end of a wire in a nice box that you put somewhere. That goes back eventually to a flip-flop. Curiously, the output from the demodulator is connected to the clock signal of the flip-flop. This causes the output to be passed through back to GPIO pin 5 of the bt878a chip, which then triggers an interrupt. I’m not clear on how it triggers an interrupt from the datasheet (it’s not GPINT, and in fact it doesn’t seem to set any status bits in INT_MASK). You then prod GPIO pin 4 which is connected to the reset line of the flip-flop. That in fact clears the interrupt, and off you go again. So basically the interrupts give you the edges of the IR pulse train.

With that information, you then have a driver time the interrupts. The supplied remote is RC5, and the edge transition bit pattern can be converted directly back to an RC5 code. Actually works quite nicely, and with a bit of tuning I have it feeling nice and smooth. Obviously the driver isn’t limited to just that remote, all you need to do is cook your own keymap. Since, weirdly, the remote seems to have the RC5 address set to TV, it interferes with my TV. So I cooked myself a keymap for another remote that I have (a Hauppauge grey remote, which is RC5 as well), and I use that instead. This nasty interrupt method seems to feel much smoother than polling the on-board IR thing on the PVR cards, although I will be working on fixing that.

I posted a linux driver to the video4linux list (one that uses the input layer, like ir-kbd-gpio or ir-kbd-i2c). You can probably find the patch in list archives (or if you are interested and can’t find it — ask and I’ll send it to you).

Anyway, it was a bit of fun and that’s my second remote control driver. I think I quite like playing with hardware.

Filed Under (Random) by mark on 28-08-2005

Final update
I have forward ported the code to the latest lirc release, fixed some bugs, and the i2c fixes have been merged into the ivtv-0.4.2. Please see this separate post, where I have removed all of the irrelevant history and only describe steps that are currently useful.

All new update
I’ve updated the ‘firmware’ file to include the new codesets recently released by hauppauge. This was a bit of a pain, as I had to disassemble my myth box to do it (bah, closed source). I hope they don’t do this too often in future! If anyone has got a spare PVR-150 or is willing to stick it in a windows machine and run wacky software then that would be nice the next time around :) Alternatively you can bribe me I guess!

Quick update: I have been working with the ivtv driver to see if I can resolve the root cause of the problem with the IR chip locking up and needing to be reset. I think that I have done so, but I can’t be sure until I get more testing feedback. I’ve updated the howto below to point at this post which contains the details of how to install the patch to ivtv. If you’ve followed all of the steps below and are still experiencing intermittent remote drop-out issues (as a couple of people have noted), you might want to try this out. I realise that this is quite a long and complicated procedure, so I’m working towards getting all fixes into the release versions of ivtv & lirc eventually. Obviously I’ll keep this updated as things progress. Slight update: up to about 28 hours now with no remote issues. Very encouraging. Further update: it works. Has been for months, and for others too. It’s in the 0.4.2 candidate release, so hopefully this issue should just die at some point. It also looks like hauppauge have put this in their mainline driver, which was updated some time in October. I’ve not disassembled it to check yet though :)

I’ve uploaded a slightly newer patch/package correcting two bugs. The first is not to reset the IR chip if you try and send an invalid code (by messing with lircd.conf!) since this clearly isn’t necessary. The second is a stack overrun bug spotted by Pascal Brisset, which could have had nasty consequences. I guess since I’ve not seen it the way I use lirc it’s only sending one code at a time to the driver. btw, this stuff is in my SVN repository, under trunk/3rdparty/lirc, if you want to do development on it.

The old stuff
OK, I’ve been playing with this off the back of the basic problem that seems to have been encountered by a lot of people, myself included — after a while the IR chip on the PVR-150 locks up, and hence the remote/blaster stop responding. (This happens with release versions of lirc/ivtv, so is definitely not related to what I’ve been doing with the IR blaster). This problem is pretty difficult to reproduce (i.e. I can’t do it on demand), but only seems to happen if you are using the encoder on the card at the time. I tried reproducing it with the Windows software, but I couldn’t manage it. I might not have tried hard enough; google search show that quite a few people have got this to fail, or it might be that hauppauge fixed the problem at some point. I don’t really know anything about this chip (it’s some software running on a fairly general purpose CPU, so it would be hard to without explicit information from hauppauge or some reverse engineering of that code, which is beyond me at this point in time), so I’m not sure why it happens. If anyone from hauppauge wants to step up and provide me with some information I’d be very happy!

There does exist a sort of solution for this: ivtvctl provides an ioctl that resets the IR chip on the card. Some people have this running every minute from a cron script, whilst that’s not a great solution it is a workable one. My issue with that is if you reset the IR chip, you have to send the boot block again to the IR blaster, so if I’d adopted the cron script I wouldn’t be able to change channel any more.

So after all that, I’ve adopted a delightful solution — detecting when the IR chip has locked up (i.e. stopped responding to i2c traffic), and resetting it in the driver. It’s messy, but it works, and as I do it in the driver, I can transparently retry IR blaster sends, which saves me from recording the wrong channel. Anyway, that was a bit long, so here’s a changelog:

- Split the code out into a new lirc driver, lirc_pvr150.
- TX and RX portions are treated as a single device. This makes good sense as you then only have one lirc socket that supports both receiving and sending (if the TX hardware is actually found).
- The RX device is ‘polled’ in the same way that the windows driver appears to. This is a good place to detect whether it has died or not.
- IR device is reset on failure and operations are retried (needs ivtv patch)
- A local copy of the ‘firmware’ data is made and release_firmware() is called. This prevents a hotplug timeout that I used to see (the script expects that firmware is released when loading is finished).
- Merge with lirc-0.72 release (rather than pre2)
- Module is reference counted properly
- Module has parameters disable_rx and disable_tx. You can use, e.g. modprobe lirc_pvr150 disable_tx=1 if you are not interested in setting up the IR blaster device.

Here’s the updated HOWTO for this version:

1. Get the pre-patched lirc 0.7.2 (or just the patch should you prefer).
2. Unpack the patched lirc (or apply the patch)

cd /usr/src
tar xfj lirc-0.7.2pvr150.tar.bz2
cd lirc-0.7.2pvr150

TV card
h – Hauppauge PVR-150 TV card (note: _NOT_ ‘g – Hauppauge TV card’)
Save configuration & run configure


make && make install.

3. Note that this step is no longer necessary with ivtv >= 0.4.0 (the current stable release series), which is recommended — the export symbol patch was included. Skip right on to step 5 if you’ve grabbed the latest stable release, or go to 3a if you are having ‘remote stops responding’ issues (doesn’t seem to affect everyone).
3a As an alternative to the below, see this post. Don’t do both. I’ll update this if that patch proves to solve problems for people. I am hoping it might, it’s working very well for me so far.
3b Patch your ivtv driver and rebuild it. This step is necessary to be able to reset the IR chip. Download the patch here. Note that this patch does not apply to some versions of ivtv. I’m currently using 0.3.7a, so it should apply cleanly to that. If you want to use a later driver, the change is actually pretty trivial, we just need to export an extra symbol. In drivers/ivtv-driver.c, you need to add:

#include “ivtv-gpio.h”

near the top of the file after the rest of the #include lines, and


right at the bottom of the file where all the other EXPORT_SYMBOL lines are. All this does is export that symbol for use by other modules, it does not change the functionality of the driver in any way. If you have trouble with this then drop me a line and I will create a patch for the version of ivtv you want to use.

cd /patch-to-ivtv-driver (e.g. cd /usr/src/ivtv-0.3.7a)
cd driver
patch < ivtv-driver.c.patch
cd ..
make && make install

4. Unload & reload the new ivtv driver. If lucky, rmmod ivtv && modprobe ivtv, otherwise either
kill everything that might be using the driver/unload all dependent modules or just reboot if you can’t be bothered with that.

5. IR blaster only: Now you need the ‘firmware’. This is
a set of data blocks that correspond to those generated by the windows software. This goes in /usr/lib/hotplug/firmware on my
debian system. Depending on your system this might also be /usr/local/lib/firmware, /lib/firmware or /lib/modules.

Note that the entire firmware is kept in memory (currently 200K) so this makes the driver quite large.
(I have no plans to sort this out, memory is cheap).

6. Check everything is working so far:

modprobe lirc_dev debug=1 && modprobe lirc_pvr150 debug=1

Check the syslog output. This should report something like:

Aug 28 02:09:11 soapbox kernel: lirc_pvr150: chip found with RX and TX
Aug 28 02:09:11 soapbox kernel: ivtv: i2c attach [client=Hauppauge PVR150 IR RX,
Aug 28 02:09:11 soapbox kernel: ivtv: i2c attach [client=Hauppauge PVR150 IR TX,
Aug 28 02:09:11 soapbox kernel: lirc_dev: lirc_register_plugin: sample_rate: 0
Aug 28 02:09:11 soapbox udev[5221]: creating device node ‘/dev/lirc0′
Aug 28 02:09:11 soapbox kernel: lirc_pvr150: firmware of size 20927 loaded
Aug 28 02:09:11 soapbox kernel: lirc_pvr150: 575 codesets loaded
Aug 28 02:09:11 soapbox kernel: lirc_pvr150: Hauppauge PVR-150 IR blaster: firmware version 1.3.0

This means that the driver has detected and initialised the IR blaster hardware — if you don’t see that then let me know.

7. You need to configure lircd, and find out which codeset you are going to be using. The easiest way is to start
with this configuration file which contains
key definitions for everything in the database.

8. Start lircd. Note: if you are using a static /dev, you may need to make a device for lirc. If unsure, once you have verified that the module has been loaded ok, run ls -l /dev/lirc*.
If you don’t see a /dev/lirc0 or similar, then try mknod /dev/lirc0 c 61 0 if the steps below fail.

modprobe lirc_dev && modprobe lirc_pvr150 debug=1
lircd --device=/dev/lirc0

9. You can now check if the remote is working using irw. Run this, and press buttons on the remote. You should see some output like

0000000000001795 00 Down Hauppauge_350.

10. Next, for the ir blaster you need to work out which codeset to use, this is the tricky bit. For this I have
send_power_new, a script that just sends the power
command in every single codeset (575 at present).

You need to stick the IR blaster on the IR receiver of box that you intend to control, being quite careful to position
it correctly — it has a very short range and took me a couple of goes to get right. Some people have reported needing
to have the device attached back to front.

If you can’t get this to work:

a) My software doesn’t work properly
b) The device you are trying to control is not supported (please check against the Windows stuff if possible)
c) You didn’t get the device in the right place — did I mention it was touchy?

11. Once you know which codeset you want you can go and delete all of the rest from lircd.conf. They are named “XXX_key”
so should be pretty easy to find. I also gave the keys standard names (0-9).

12. To get mythtv to work, configure a channel change script for your device. There’s one
here that should work out of the box if you
rename the number keys.

That’s it, good luck!

Filed Under (Random) by mark on 19-08-2005

I have updated the patch with the following changes:

- Use schedule_timeout not mdelay as mdelay is a busy loop. This fixes high CPU usage during IR sending.
- Drop out of the polling loop when the chip is ready rather than after 1s (can’t believe I didn’t notice this one!). This greatly speeds sending.

As a consequence of the second point, I recommend that you fiddle with the ‘gap’ parameter in lircd.conf. The gap parameter is in microseconds. For my particular cable box, I’ve found that 1/3s (~333,333us) works well. The minimum time is until the chip is ready to send, this works out roughly at about 50-200ms. The minimum delay is fixed at 50ms (avoids wasting CPU; go hack the code if you want to change this), and the maximum wait is 1s.