tag:blogger.com,1999:blog-329222472024-03-21T08:02:02.196-07:00Over Macho GrandeCoreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.comBlogger28125tag:blogger.com,1999:blog-32922247.post-3302783975112385242017-09-23T11:22:00.001-07:002017-09-23T11:22:32.303-07:00Building Amazon Linux RPMs with Fedora MockFedora's 'mock' tool provides a much more convenient way to build RPMs than using 'rpmbuild'. It creates a chroot environment for your target OS and will install required dependencies.<br />
<br />
I've been running into limitations on the version of CollectD that ships with Amazon Linux, so I thought it wouldn't be that difficult to use mock to build an updated version complete with some missing plugins. I knew that it generally worked, as I was able to use the specfile from the project to build for EPEL6 using Fedora 26. Man, was I wrong about how easy it would be for Amazon Linux.<br />
<br />
I won't go into details here, but it's worth mentioning that the CollectD project documentation calls out that the specfile in their contrib directory is generally out of date. That is 100% correct, so you'll need to budget some time for tweaking it.<br />
<br />
The first issue is that there are some packages in EPEL that can't be installed in Amazon Linux. The most aggravating are libyajl/libyajl-devel, both of which are required for the curl_json plugin in CollectD.<br />
<br />
Working around that particular issue required grabbing the SRPM from EPEL, extracting the contents, and building for Amazon Linux. Then I had to figure out how to pull those packages into my mock workflow. Luckily, that was as easy as putting them into a local directory and calling 'createrepo'.<br />
<br />
The next problem was that Amazon's general Python 2.6 RPM conflicts with another RPM from EPEL (python-rpm-macros). Thanks to <a href="https://www.reancloud.com/blog/extend-amazon-linux-with-software-from-rpm-based-linux-distributions/">these kind folks</a>, I was able to figure out that I could exclude packages. Unfortunately, 'python-rpm-macros' isn't a direct dependency, so I had to run mock a few times to figure out how to blacklist the correct packages.<br />
<br />
After some experimentation, here's my mock config file for Amazon Linux 2017.03. I'm running in the us-west-2 region, so if you're running elsewhere you'll obviously need to change the region in the file.<br />
<br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
config_opts['root'] = 'amazon-2017.03-x86_64'</blockquote>
<blockquote class="tr_bq">
config_opts['target_arch'] = 'x86_64'</blockquote>
<blockquote class="tr_bq">
config_opts['legal_host_arches'] = ('x86_64',)</blockquote>
<blockquote class="tr_bq">
config_opts['chroot_setup_cmd'] = 'install @buildsys-build'</blockquote>
<blockquote class="tr_bq">
#config_opts['dist'] = 'el6' # only useful for --resultdir variable subst</blockquote>
<blockquote class="tr_bq">
config_opts['releasever'] = '2017.03'</blockquote>
<blockquote class="tr_bq">
config_opts['use_nspawn'] = False</blockquote>
<blockquote class="tr_bq">
config_opts['awsdomain'] = 'amazonaws.com'</blockquote>
<blockquote class="tr_bq">
config_opts['awsregion'] = 'us-west-2'</blockquote>
<blockquote class="tr_bq">
<br /></blockquote>
<blockquote class="tr_bq">
config_opts['yum.conf'] = """</blockquote>
<blockquote class="tr_bq">
[main]</blockquote>
<blockquote class="tr_bq">
keepcache=1</blockquote>
<blockquote class="tr_bq">
debuglevel=2</blockquote>
<blockquote class="tr_bq">
reposdir=/dev/null</blockquote>
<blockquote class="tr_bq">
logfile=/var/log/yum.log</blockquote>
<blockquote class="tr_bq">
retries=20</blockquote>
<blockquote class="tr_bq">
obsoletes=1</blockquote>
<blockquote class="tr_bq">
gpgcheck=0</blockquote>
<blockquote class="tr_bq">
assumeyes=1</blockquote>
<blockquote class="tr_bq">
syslog_ident=mock</blockquote>
<blockquote class="tr_bq">
syslog_device=</blockquote>
<blockquote class="tr_bq">
mdpolicy=group:primary</blockquote>
<blockquote class="tr_bq">
best=1</blockquote>
<blockquote class="tr_bq">
<br /></blockquote>
<blockquote class="tr_bq">
# repos</blockquote>
<blockquote class="tr_bq">
[amzn-main]</blockquote>
<blockquote class="tr_bq">
name=amzn-main-Base</blockquote>
<blockquote class="tr_bq">
mirrorlist=http://repo.us-west-2.amazonaws.com/$releasever/main/mirror.list</blockquote>
<blockquote class="tr_bq">
mirrorlist_expire=300</blockquote>
<blockquote class="tr_bq">
metadata_expire=300</blockquote>
<blockquote class="tr_bq">
priority=10</blockquote>
<blockquote class="tr_bq">
failovermethod=priority</blockquote>
<blockquote class="tr_bq">
fastestmirror_enabled=0</blockquote>
<blockquote class="tr_bq">
gpgcheck=1</blockquote>
<blockquote class="tr_bq">
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-amazon-ga</blockquote>
<blockquote class="tr_bq">
enabled=1</blockquote>
<blockquote class="tr_bq">
retries=5</blockquote>
<blockquote class="tr_bq">
timeout=10</blockquote>
<blockquote class="tr_bq">
report_instanceid=yes</blockquote>
<blockquote class="tr_bq">
<br /></blockquote>
<blockquote class="tr_bq">
[amzn-main-debuginfo]</blockquote>
<blockquote class="tr_bq">
name=amzn-main-debuginfo</blockquote>
<blockquote class="tr_bq">
mirrorlist=http://repo.us-west-2.amazonaws.com/$releasever/main/debuginfo/mirror.list</blockquote>
<blockquote class="tr_bq">
mirrorlist_expire=300</blockquote>
<blockquote class="tr_bq">
metadata_expire=300</blockquote>
<blockquote class="tr_bq">
priority=10</blockquote>
<blockquote class="tr_bq">
failovermethod=priority</blockquote>
<blockquote class="tr_bq">
fastestmirror_enabled=0</blockquote>
<blockquote class="tr_bq">
gpgcheck=1</blockquote>
<blockquote class="tr_bq">
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-amazon-ga</blockquote>
<blockquote class="tr_bq">
enabled=0</blockquote>
<blockquote class="tr_bq">
retries=5</blockquote>
<blockquote class="tr_bq">
timeout=10</blockquote>
<blockquote class="tr_bq">
report_instanceid=yes</blockquote>
<blockquote class="tr_bq">
<br /></blockquote>
<blockquote class="tr_bq">
[amzn-main-source]</blockquote>
<blockquote class="tr_bq">
name=amzn-main-source</blockquote>
<blockquote class="tr_bq">
mirrorlist=http://repo.us-west-2.amazonaws.com/$releasever/main/SRPMS/mirror.list</blockquote>
<blockquote class="tr_bq">
mirrorlist_expire=300</blockquote>
<blockquote class="tr_bq">
metadata_expire=300</blockquote>
<blockquote class="tr_bq">
priority=10</blockquote>
<blockquote class="tr_bq">
failovermethod=priority</blockquote>
<blockquote class="tr_bq">
fastestmirror_enabled=0</blockquote>
<blockquote class="tr_bq">
gpgcheck=1</blockquote>
<blockquote class="tr_bq">
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-amazon-ga</blockquote>
<blockquote class="tr_bq">
enabled=0</blockquote>
<blockquote class="tr_bq">
retries=5</blockquote>
<blockquote class="tr_bq">
timeout=10</blockquote>
<blockquote class="tr_bq">
report_instanceid=yes</blockquote>
<blockquote class="tr_bq">
<br /></blockquote>
<blockquote class="tr_bq">
[in-progress]</blockquote>
<blockquote class="tr_bq">
name=in-progress</blockquote>
<blockquote class="tr_bq">
baseurl=file:///home/fedora/in-progress</blockquote>
<blockquote class="tr_bq">
<br /></blockquote>
<blockquote class="tr_bq">
[epel]</blockquote>
<blockquote class="tr_bq">
name=epel</blockquote>
<blockquote class="tr_bq">
mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=epel-6&arch=x86_64</blockquote>
<blockquote class="tr_bq">
failovermethod=priority</blockquote>
<blockquote class="tr_bq">
gpgkey=file:///usr/share/distribution-gpg-keys/epel/RPM-GPG-KEY-EPEL-6</blockquote>
<blockquote class="tr_bq">
gpgcheck=1</blockquote>
<blockquote class="tr_bq">
exclude=python-rpm-macros,epel-rpm-macros-*</blockquote>
<blockquote class="tr_bq">
<br /></blockquote>
<blockquote class="tr_bq">
[local]</blockquote>
<blockquote class="tr_bq">
name=local</blockquote>
<blockquote class="tr_bq">
baseurl=https://kojipkgs.fedoraproject.org/repos/dist-6E-epel-build/latest/x86_64/</blockquote>
<blockquote class="tr_bq">
cost=2000</blockquote>
<blockquote class="tr_bq">
enabled=0</blockquote>
<blockquote class="tr_bq">
"""</blockquote>
</blockquote>
Building the SRPM is this:<br />
<br />
<blockquote class="tr_bq">
mock -r /etc/mock/amazon-2017.03-x86_64.cfg --buildsrpm --spec $HOME/rpmbuild/SPECS/collectd.spec --sources $HOME/rpmbuild/SOURCES</blockquote>
<br />
And then building the RPM from the SRPM is this:<br />
<br />
<blockquote class="tr_bq">
mock -r /etc/mock/amazon-2017.03-x86_64.cfg --no-clean --rebuild /var/lib/mock/amazon-2017.03-x86_64/result/collectd-5.7.2-3.amzn1.src.rpm</blockquote>
There's a couple other native packages I want to build, so at some point I plan to put all this stuff up in GitHub to make it easier to consume.<br />
<blockquote class="tr_bq">
</blockquote>
<blockquote class="tr_bq">
</blockquote>
<br />
<br />Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-82789121068539748432017-09-08T15:32:00.001-07:002017-09-08T15:32:11.354-07:00Mass updating AWS Lambda Log Group retentionAWS Lambda and I have a love/hate relationship. There is much about Lambda to like, but there are also some very sharp edges operationally.<br />
<br />
One of the cool things is that you get a new CloudWatch Log Group for every new Lambda function without any effort on your part. Less cool is that it has unlimited retention. If you haven't yet followed Yan Cui's <a href="https://hackernoon.com/centralised-logging-for-aws-lambda-b765b7ca9152">advice</a>, then you can use some Bash/CLI magic to fix retention on your existing Log Groups.<br />
<br />
First, get a list of all your default Lambda log groups:<br />
<br />
<blockquote class="tr_bq">
aws logs describe-log-groups --log-group-name-prefix "/aws/lambda" | grep logGroupName | cut -d : -f 2 | cut -d \" -f 2 > /tmp/lambda_logs</blockquote>
<br />
Read that into a Bash array:<br />
<br />
<blockquote class="tr_bq">
readarray -t log_groups < /tmp/lambda_logs</blockquote>
<br />
Then, add a 7 day retention policy to all those log groups:<br />
<br />
<blockquote class="tr_bq">
for i in "${log_groups[@]}"; do aws logs put-retention-policy --log-group-name $i --retention-in-days 7; done</blockquote>
<br />
It's a hack, but if you're going to put in the effort to do it right, then set up that CloudWatch Event/Lambda that manages it for you.Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-51030210830357720532014-08-31T03:27:00.003-07:002014-08-31T03:27:20.015-07:00MSBuild, NuGet restore, and HintPathFirst of, let me start by acknowledging that you're no longer supposed to use NuGet restore (with the .nuget) as of NuGet 2.7. Then again, to quote Mick Jagger, you can't always get what you want.<br />
<br />
The biggest problem I have with this is the "HintPath" in the project file. It took some experimentation (and a couple of wrong turns) to work out that HintPath is relative to the project directory <i>even when it's part of a multi-project solution</i>.<br />
<br />
You'll know that your HintPath is wrong when the NuGet package is installed but can't be found by the compiler. Drop into the directory containing your project and confirm that you've got a valid HintPath. One of my wrong turns was not verifying that I had a correct path (there was an extra 'lib' in the definition.<br />
<br />
You may also see this as a busted reference in Visual Studio.<br />
<br />
<br />Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-63791834522491779492013-06-24T18:47:00.002-07:002013-06-24T18:47:45.278-07:00Application backstoryI'm in the "winding down" phase of my time here in Noumea. One of my cornerstone achievements is migrating a buggy MS Access app into a shiny new MVC3 (and then MVC4) web application.<br />
<br />
I've been working on it off and on for most of my time here (maybe 18 months) and one of my co-workers asked me to document some of the design decisions. The app kind of grew organically, so I don't have a software spec to point to like I would have had at my prior engagements. I felt a series of blog posts would be an appropriate way of documenting this. It's an open source app, so the blog posts are my defense when someone digs into the app and finds some awful hack that migrated from a short-term fix to a permanent solution.<br />
<br />
I've got nothing more today, but next time I'll dig into the choice of data access layer.<br />
<br />
<a href="https://github.com/spc-ofp/tubs-web">TUBS on GitHub</a>Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-87642474393403734122012-05-11T17:14:00.002-07:002012-05-11T17:14:46.009-07:00Don't buy a Portege R830Okay, so this is something of a rant, but given my experiences with the laptop, how can I not?<br />
<br />
Backstory: Corporate IT maintains a list of acceptable portables, all of which are made by Toshiba. After looking over the list, it looked like the best choice was the Portege R830. It had the best possible CPU and it wasn't a giant clunker that looked my back-in-the-day 15 pound Inspiron 7000. As a bonus, it was available with a reasonably priced docking station.<br />
<br />
My first conception that something was wrong when, after first boot, I checked free memory. Physically, 4GB is installed, but windows says that only 2.7GB is available. The Intel HD3000 video setup is eating the rest (1.3GB!). Okay, no big deal. I can reboot and change that in the BIOS. Oops. No option. Well, maybe it's just some old drivers. Nope. Drivers are latest from Toshiba. They're not particularly new, so maybe the generic Intel ones will fix it. Damn! Toshiba has crippled the video so that it won't accept the Intel drivers.<br />
<br />
Maybe if I install more physical memory. As an aside, IT installed the 32-bit version of Win7. I know, I know. But I've got duty travel coming up, and getting the laptop within 3 months was a major achievement as it is. Asking for a wipe on day two isn't going to win me any friends.<br />
<br />
So, I dropped 8GB into the computer on the assumption that the BIOS would be smart enough to leave the 4GB that Win7 _can_ address to me, and take the rest for video, even if it is overkill. No dice: The BIOS reserves nearly 6GB of the 8GB memory for video. Really? Really?!<br />
<br />
A tech support request to Toshiba hasn't been answered. I got the obligatory "We'll get back to you within 24 hours" email, but no actual email.<br />
<br />
I wouldn't think that being able to specify video memory would be all that big of a deal, but then again, this BIOS is pretty braindead. I think I've seen more options on locked-down Pentium 1 era computers. I'm not expecting overclocking options, but at least let me use the damn thing, especially if IT has already f**ked things up by only installing half the memory I asked for and then installing the 32-bit version of Windows.<br />
<br />
<br />Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-54828140656707382892012-04-15T03:39:00.000-07:002012-04-15T03:39:11.651-07:00MVC3 Unobtrusive Validation, Bootstrap, and YouI _really_ like Twitter's Bootstrap. It saves me from making any number of questionable content design choices. I'm using it in an MVC3 project and one of the things that bugged me was the inability to get Microsoft's unobtrusive validation working with the fancy form control states in Bootstrap.<br />
<br />
Well, after looking around on StackOverflow and poking and prodding MVC, here's what I've got:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitgs_9lSh5NuQwYYVnxjLIAGVgbl-Nvvw-xQf8EMJPjuweBMWZn3bm9IzVW9Q8PkKBy-mEaFUBbuzy88yAg3LOgOHvSsKkTCkaiKFb-dZ7lHIPGhiCI2TNMeJA9XzPCzwsodvV6A/s1600/invalid.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="305" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitgs_9lSh5NuQwYYVnxjLIAGVgbl-Nvvw-xQf8EMJPjuweBMWZn3bm9IzVW9Q8PkKBy-mEaFUBbuzy88yAg3LOgOHvSsKkTCkaiKFb-dZ7lHIPGhiCI2TNMeJA9XzPCzwsodvV6A/s320/invalid.png" width="320" /></a></div>
If you correct the error, the state changes away from error and you get this:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhF6PCgJPtAPR4CsfzLP3gE4FFi0aQgIhflMOV6xyd46ttqSr8tFQdiS75jV-blLyt54eCsugeSBBEe8EV2KL0I_Qij6G-FVNlxQ6OI6rF_qCFbdcJvCIAGg9607FZ0npwg5GBmtg/s1600/valid.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhF6PCgJPtAPR4CsfzLP3gE4FFi0aQgIhflMOV6xyd46ttqSr8tFQdiS75jV-blLyt54eCsugeSBBEe8EV2KL0I_Qij6G-FVNlxQ6OI6rF_qCFbdcJvCIAGg9607FZ0npwg5GBmtg/s320/valid.png" width="263" /></a></div>
I hacked this into place by inserting a shim error placement function. It saves a handle to the original Microsoft implementation, changes the class on the nearest div with a CSS class of "control-group" and then calls the Microsoft implementation to display the actual error text.<br />
<br />
<blockquote class="tr_bq">
$(document).ready(function () {<br /> var esettings = $.data($('form')[0], 'validator').settings;<br /> // Get a handle to the original errorPlacement function<br /> var originalFunction = esettings.errorPlacement;<br /> esettings.errorPlacement = function (error, inputElement) {<br /> // Although you have access to the form via $(this),<br /> // getElementById should be efficient<br /> // NOTE: This assumes that the name and id properties are the same<br /> var id = "#" + inputElement[0].name;<br /> var controlGroup = $(id).closest("div.control-group");<br /> // error[0].innerHTML contains the validation error message<br /> if (0 == error[0].innerHTML.length) {<br /> controlGroup.removeClass("error").addClass("success");<br /> } else {<br /> controlGroup.removeClass("success").addClass("error");<br /> }<br /> // Call the original function<br /> originalFunction(error, inputElement);<br /> }<br /> });</blockquote>
There's probably a much smarter way to do this, but this works well enough for now.Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-11548336141052177152012-03-27T20:40:00.000-07:002012-03-27T20:40:02.033-07:00Initial Speech Recognition AppI'm pretty impressed with Microsoft's System.Speech API. It took less than 3 days to throw together a proof-of-concept application. The hardest part was probably coming up with the grammar -- documentation for that is pretty thin on the ground.<br />
<br />
Anyways, here's the application source code on GitHub if anyone wants a look:<br />
<a href="https://github.com/spc-ofp/ObserverLengthSampler">ObserverLengthSampler project</a><br />
<br />
If nothing else, I'd recommend it as a starting point for someone needing a number recognition SRGS grammar in an XML format.Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-26776618259652077782012-03-26T02:53:00.001-07:002012-03-26T02:53:02.400-07:00Speech Recognition<div><p>I just got a short side project @ work: Use speech recognition to capture fish species and length from tuna observer samples. I'm going to try the win7 version first but I'll probably wind up giving sphinx a try too.</p>
</div>Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0N'Géa, N'Géa-22.289774 166.45486tag:blogger.com,1999:blog-32922247.post-70823638603687196862011-11-27T01:12:00.001-07:002011-11-27T01:18:11.430-07:00Gingerbread reduxI've been running GingerStreak of late and I'm not best pleased with the results. So I thought I'd try my hand at
building Gingerbread for the Streak. Dell has helped out by actually releasing a working kernel.
Just so I remember, here's how I put myself onto the latest released msm8660 build (gingerbread_house @ CodeAurora):<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">repo forall -c git checkout M8660AAABQNLZA3620 </span><br />
<br />
Initially I just moved the original externals to a .aosp directory (e.g. bluetooth -> bluetooth.aosp)
so that I could diff them, but the default build process still wants to build 'em.
Looks like I'll have to move it outside the tree.Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-82571832723938933062011-10-14T20:24:00.000-07:002011-10-14T20:24:13.175-07:00Python and libpuzzleAs much as I've dogged on Python in the past (significant whitespace, really?), I've got to admit that it's<br />
got some cool features too.<br />
<br />
For example, I'm playing with <a href="https://github.com/jedisct1/libpuzzle">libpuzzle</a> (a library for visually comparing images). It has a command line utility and a C and PHP API. Unfortunately, the CLI utility doesn't allow one to dump the raw comparison vector, and it's a PITA to write C just to play with a library.<br />
<br />
Python's native "ctypes" to the rescue!<br />
<br />
<pre>from ctypes import *
class PuzzleCvec(Structure):
_fields_ = [("sizeof_vec", c_size_t),
("vec", c_char_p)]
class PuzzleCompressedCvec(Structure):
_fields_ = [("sizeof_compressed_vec", c_size_t),
("vec", c_char_p)]
class PuzzleContext(Structure):
_fields_ = [("puzzle_max_width", c_uint),
("puzzle_max_height", c_uint),
("puzzle_lambdas", c_uint),
("puzzle_p_ratio", c_double),
("puzzle_noise_cutoff", c_double),
("puzzle_contrast_barrier_for_cropping", c_double),
("puzzle_max_cropping_ratio", c_double),
("puzzle_enable_autocrop", c_int),
("magic", c_ulong)]
libpuzzle = CDLL('libpuzzle.so')
context = PuzzleContext()
cvec = PuzzleCvec()
ccvec = PuzzleCompressedCvec()
libpuzzle.puzzle_init_context(byref(context))
libpuzzle.puzzle_init_cvec(byref(context), byref(cvec))
libpuzzle.puzzle_init_compressed_cvec(byref(context), byref(ccvec))
fileName = '/home/corey/page1.png'
retval = libpuzzle.puzzle_fill_cvec_from_file(byref(context), byref(cvec), fileName)
print "fill_cvec returned %d" % (retval)
print "Vector length: %d" % (cvec.sizeof_vec)
retval = libpuzzle.puzzle_compress_cvec(byref(context), byref(ccvec), byref(cvec))
print "compress_cvec returned %d" % (retval)
print "Compressed vector length: %d" % (ccvec.sizeof_compressed_vec)
print "Vector: %s" % (ccvec.vec)
# Clean up
libpuzzle.puzzle_free_cvec(byref(context), byref(cvec))
libpuzzle.puzzle_free_context(byref(context))
</pre>
<br />
Now I can play around with the library from Python, for example dumping the vector for each image into SQLite.Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com1tag:blogger.com,1999:blog-32922247.post-65181242487868986112011-09-09T00:06:00.000-07:002011-09-09T00:06:17.008-07:00I guess Python isn't so bad after all...Not wanting to hassle with learning OpenCV and fighting with an edit-compile-execute environment, I decided to use my OpenCV project as an excuse to play around with Python.<br />
<br />
I'm still a serious beginner, but I'm beginning to understand why it gets the use it does.<br />
<br />
Anyhow, it only took a couple of days to integrate Tesseract OCR, PIL, and OpenCV such that I could open multi-frame TIFF images, perform some basic feature detection, and then use the output of feature detection to focus on a specific region for OCR.<br />
<br />
I will admit to having a few false starts. The first was that I used an older (C++) tutorial that was using some deprecated features of OpenCV and ignoring some other features. For example, the tutorial was using Hough Line detection to find squares on a printed page. In order to get to that point there was thresholding, dilating, eroding, inversion, flood filling and so on. Even then I wasn't getting the correct results.<br />
<br />
When I started over, I was using the old SWIG based interface instead of the newer stuff in the cv2 namespace. Having used the newer stuff I've got to say it's much easier, especially when you need to move back and forth between OpenCV and PIL -- both libraries understand Numpy arrays.Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-56338837749730052822011-09-06T00:35:00.001-07:002011-09-06T00:35:45.919-07:00Android on holdSince the Android git repository is offline, I'm having to find something else to occupy my time.<br />
<br />
I've always wanted to learn more about OpenCV, so I'm working on a new project using that. At work we have piles and piles of paper forms that have been filled out by hand. I've already done some trial and error work and determined that both OpenCV and PIL can clean up the scanned copies enough that I can do OCR on the printed portion of the forms using Tesseract OCR. This doesn't get any of the dynamic data, but it does allow me to identify what type of form it is.<br />
<br />
Running OCR on an entire document takes some time, so it would be better to grab smaller regions of interest and only OCR them. Some of the interesting challenges are that some forms are portrait while others are landscape. I'd also like to handle the case where a form was fed into the scanner upside down.Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-1330751540295019472011-03-27T16:50:00.000-07:002011-03-27T16:50:37.868-07:00If it builds, ship it...I've finally gotten a working build. It turns out what I really needed was to get up @ 0630, take a shower, and just plug away.<br />
<br />
Anyhow, the source is up on github: <a href="https://github.com/coredog64">https://github.com/coredog64</a><br />
<br />
I still need to get it to build an image for the Streak before I can actually test it.Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com1tag:blogger.com,1999:blog-32922247.post-73084987312386794462011-03-20T21:19:00.000-07:002011-03-20T21:19:13.651-07:00More progressOn St. Patrick's Day, Dell released an early alpha version of a Streak gingerbread kernel. They're working from a CodeAurora drop from last year and they're using Broadcom provided brcm4325 drivers instead of the brcm4329 drivers that support both chipsets.<br />
<br />
So, I merged the Dell alpha into the current CodeAurora tree and have something that almost builds.<br />
<br />
The last piece fell into place today when I found a streak device tree for froyo. There have been some breaking changes in the build files between froyo and gingerbread, so I need to touch all the make files to get any further.<br />
<br />
Still, I might actually have my own gingerbread ROM right around the time Google drops Honeycomb and Dell releases their own version of gingerbread...Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-6744662077612944252011-03-12T22:15:00.000-07:002011-03-12T22:15:04.551-07:00Almost thereAll the Dell files compile correctly. All I have to do now is work out the config changes so that<br />
the correct object files are built and linked into the final kernel.Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-37666114297713947992011-03-11T21:21:00.000-07:002011-03-11T21:21:00.054-07:00PEBKACIt turns out the reason it wasn't building the object files into the built-in was that I had neglected to copy the source files for them into the CodeAurora tree.<br />
<br />
I finished merging the Dell and CodeAurora board files into an appropriate version for the Streak. It still doesn't finish building, but I'm much closer.Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-1174905388830965572011-03-10T21:06:00.000-07:002011-03-10T21:06:18.054-07:00RoadblocksMostly roadblocks tonight, although I did learn how to remove the "-Werror" flag from the set of flags used to compile a specific object.<br />
<br />
As of right now, I'm not understanding why my driver object files aren't being pulled into the built-in driver object file. I suspect I'm going to have to dig deeper into the top level makefile.Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-90736499641423526032011-03-05T23:21:00.000-07:002011-03-05T23:21:00.037-07:00Getting closerDiscovered that the "toucan" config is distinct from the "austin" config and so can dump a bunch of<br />
drivers.<br />
<br />
Unfortunately it looks like I have to merge the Dell board config (board-qsd8x50_austin.c) with the latest generic board config from CodeAurora (board-qsd8x50.c).<br />
<br />
Here's where I'm at so far:<br />
<br />
<pre>corey@patches:~/msm$ git status
# On branch gingerbread_rel
# Changed but not updated:
# (use "git add <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
#
# modified: arch/arm/mach-msm/Kconfig
# modified: arch/arm/mach-msm/Makefile
# modified: arch/arm/mach-msm/include/mach/board.h
# modified: arch/arm/mach-msm/include/mach/camera.h
# modified: drivers/input/keyboard/Kconfig
# modified: drivers/input/keyboard/Makefile
# modified: drivers/input/misc/Makefile
# modified: drivers/input/touchscreen/Kconfig
# modified: drivers/input/touchscreen/Makefile
# modified: drivers/leds/Kconfig
# modified: drivers/leds/Makefile
# modified: drivers/media/video/msm/Kconfig
# modified: drivers/media/video/msm/Makefile
# modified: drivers/media/video/msm/msm_camera.c
# modified: drivers/media/video/msm/msm_io8x.c
# modified: drivers/media/video/msm/msm_vfe8x_proc.c
# modified: drivers/misc/Makefile
# modified: drivers/power/Kconfig
# modified: drivers/serial/msm_serial.c
#
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# arch/arm/configs/austin-perf_defconfig
# arch/arm/configs/current_defconfig
# arch/arm/mach-msm/board-msm8x60-smmu.c
# arch/arm/mach-msm/board-qsd8x50_austin.c
# arch/arm/mach-msm/bootloader.c
# arch/arm/mach-msm/bt-rfkill.c
# arch/arm/mach-msm/cust_mproc.c
# arch/arm/mach-msm/filter.c
# arch/arm/mach-msm/include/mach/auo_touch.h
# arch/arm/mach-msm/include/mach/austin_hwid.h
# arch/arm/mach-msm/include/mach/bcom_fm.h
# arch/arm/mach-msm/include/mach/cover.h
# arch/arm/mach-msm/include/mach/custmproc.h
# arch/arm/mach-msm/include/mach/gsensor.h
# arch/arm/mach-msm/include/mach/lsensor.h
# arch/arm/mach-msm/include/mach/qsd_battery.h
# arch/arm/mach-msm/include/mach/sensors_daemon.h
# arch/arm/mach-msm/smd_rpc_sym.c
# arch/arm/mach-msm/smmu_driver.c
# arch/arm/mach-msm/smmu_driver_enum.c
# arch/arm/mach-msm/tlmm-msm7200a.c
# arch/arm/mach-msm/tlmm-msm8660.c
# arch/arm/mach-msm/tlmm-msm8660.h
# drivers/input/keyboard/i2ccapkbd.c
# drivers/input/keyboard/mutekey.c
# drivers/input/misc/cover.c
# drivers/input/misc/lsensor_austin.c
# drivers/input/misc/lsensor_austin.h
# drivers/input/misc/psensor-cm3603.c
# drivers/input/touchscreen/auo_touch.c
# drivers/media/video/msm/ov5642.c
# drivers/media/video/msm/ov5642.h
# drivers/media/video/msm/ov5642_1C.c
# drivers/media/video/msm/ov5642_1D.c
# drivers/media/video/msm/ov5642_af.h
# drivers/media/video/msm/ov5642_comm.c
# drivers/media/video/msm/ov7690.c
# drivers/media/video/msm/ov7690.h
# drivers/media/video/msm/ov7690_comm.c
# drivers/misc/pmem_kernel_test.c
# drivers/misc/reset/
# drivers/power/qsd_battery.c
# drivers/power/qsd_battery.h
# drivers/video/msm_austin/
</file></file></file></pre>Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-45796700424176902072011-03-03T21:04:00.002-07:002011-03-03T21:04:48.132-07:00No gingerbread work tonightI spent tonight working on a super secret project. Hopefully I'll be able to say more in a few weeks.Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-63147194581245220072011-03-01T19:11:00.000-07:002011-03-01T19:11:25.024-07:00Some driver progress...I diff'd the Dell source release against 2.6.35 from CodeAurora. A lot of the changes were noise -- Intel wireless driver headers that got renamed, or Zaurus support that's gone away.<br />
<br />
What I'm left with is stuff in the following directories:<br />
/drivers/i2c/chips<br />
/drivers/input/keyboard<br />
/drivers/input/misc<br />
/drivers/input/touchscreen<br />
/drivers/leds<br />
/drivers/media/video<br />
/drivers/media/video/msm<br />
/drivers/misc<br />
/drivers/misc/reset<br />
/drivers/mmc/host<br />
/drivers/mtd/devices<br />
/drivers/net/msm_rmnet_*<br />
/drivers/power/qsd_batter.*<br />
/drivers/staging/android<br />
/drivers/video/msm_austin<br />
/drivers/video/msm<br />
/drivers/video/msm_toucan<br />
/drivers/watchdog<br />
<br />
Another help is that someone else posted a mention that the Streak's codename was "Austin", so a grep through the Dell source tree turned up a bunch of files that have Streak specific changes.Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-12878016091574569192011-02-27T21:57:00.000-07:002011-02-27T21:57:15.348-07:00Starting my Streak Gingerbread portWhile I appreciate all the hard work DJ_Steve puts into StreakDroid, it's just too buggy for me. So I'm going to try my hand at jumping right to Gingerbread.<br />
<br />
My starting point is the MSM/QSD code that Qualcomm publishes at CodeAurora. Clone the<br />
kernel repo using the following command:<br />
<br />
git clone git://codeaurora.org/kernel/msm.git<br />
<br />
Next, I want to change the working master to the Gingerbread kernel:<br />
<br />
git checkout -b gingerbread_rel origin/gingerbread_rel<br />
<br />
My next step is to pull in Dell's changes from their 3.09 release. Stay tuned!Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-31405551244511952812007-08-04T19:57:00.000-07:002007-08-04T20:16:36.668-07:00Flash video to Disney MixMaxI'm a sucker for electronic kid gadgets, as long as they're on clearance. In our house we've got several Juice Boxes, a VideoNow PVD, and even a Disney MixMax video player. The MixMax hits the sweet spot in that it uses cheap SD memory and uses WMV as it's video format.<br /><br />My kids love to watch <a href="http://pbskids.org/wordgirl/">WordGirl</a>. Unfortunately, there's no DVD of these shorts. The friendly folks over at PBS Kids have (intentionally or otherwise) put all the Flash video clips online in a directory of their web server (<a href="http://pbskids.org/wordgirl/content/video">http://pbskids.org/wordgirl/content/video</a>).<br /><br />It took me a couple of days, but I can now share how I converted the files so they work on the MixMax.<br /><br />My initial experiments centered on using ffmpeg on Linux to convert to a common format. It worked out okay on some of the videos, but not all of them. ffmpeg can encode in WMV/WMA, so then I tried that. Again, it worked for some videos but not all of them. As my Linux box also happens to be my Windows box, rebooting soon became tiresome and I moved to a Win32 port of ffmpeg.<br /><br />Here's the ffmpeg command line I used to create a "raw" video file.<br /><br />ffmpeg -i x.flv -b 512k -vcodec huffyuv -acodec pcm_u8 -r 24 x.avi<br /><br />(Three notes: One, 'huffyuv' requires ffdshow to be installed. Two, this codec was chosen through trial and error, not some intrinsic "goodness" for the task at hand. Three, these files still need to be converted to the correct WMV format.)<br /><br />Converting video files to the appropriate size, bitrate, etc. for the MixMax is a pain if you use the standard encoder (i.e. what the MixMax instructions recommend.) As I'm more than comfortable on the commandline I use the WSH script that comes with the encoder.<br /><br />Here's the command line that does the WMV conversion:<br /><br />cscript "C:\Program Files\Windows Media Components\Encoder\WMCmd.vbs" -input x.avi -output x.wmv -loadprofile "C:\Program Files\Windows Media Components\Encoder\Profiles\mixmax.prx"<br /><br />(mixmax.prx is not a standard profile -- Google is your friend here)Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-40177301171814543012007-04-20T16:53:00.000-07:002007-04-20T16:55:00.402-07:00What a warning message!I'm reinstalling Solaris 9 OE and "smpatch" is running on the SB1000 right now.<br />The following error message appeared on the screen:<br /><br /><em>CRITICAL: The patch 112838-12 makes the kernel unstable, is obsolete or is forbidden by policy.</em><br /><em></em><br />How's that for automated patch management?Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-13353858708739190422007-04-08T21:25:00.000-07:002007-04-08T21:33:25.213-07:00MIT Kerberos for SolarisI compiled the latest version of MIT Kerberos for Solaris tonight. All in all not too painful. I hadn't done any compilation since December, so I forgot that it's best to use bash instead of tcsh as my configuration shell.<br /><br />For future reference, here's the configure options:<br /><br /><em>./configure --prefix=/usr/local --without-krb4 --with-tcl=/usr/sfw --enable-shared</em>Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0tag:blogger.com,1999:blog-32922247.post-9035956639202430362007-03-11T10:20:00.000-07:002007-03-11T10:24:22.222-07:00SharePoint via Java (redux)Not long after I implemented a UNIX to SharePoint Document Library<br />transfer service at work we moved to a native Kerberos domain at work.<br /><br />The upside is that we're using Kerberos and can play with stuff<br />like <a href="http://modauthkerb.sourceforge.net/">mod_auth_kerb</a>. The downside is that the current version<br />of HttpClient doesn't work in an AD/Kerberos environment.Coreyhttp://www.blogger.com/profile/07735573947651089652noreply@blogger.com0