{"id":70,"date":"2010-11-10T00:05:49","date_gmt":"2010-11-10T00:05:49","guid":{"rendered":"http:\/\/blog.danplanet.com\/wordpress\/?p=70"},"modified":"2012-02-09T16:11:19","modified_gmt":"2012-02-09T16:11:19","slug":"a-better-way-to-process-binary-data-in-python","status":"publish","type":"post","link":"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/","title":{"rendered":"A better way to process binary data in Python"},"content":{"rendered":"

One of my side-projects is CHIRP<\/a>, which is an application for programming the memory contents of various radios.\u00a0 Like other projects I’ve done where I want to cater to (or at least, not exclude) people running Windows, it’s written in Python.\u00a0 Not only is Python my newfangled language of choice for high-level geekiness, but it’s also a language that provides me a substantial amount of platform isolation and I’m comfortable deploying applications based on it to Windows.\u00a0 I really have no desire to learn how to write “real” Windows applications, so if it weren’t as easy as it is, I wouldn’t do it.<\/p>\n

Python is really great for a lot of things, but processing binary data is not what I would consider one of them.\u00a0 CHIRP needs to download a binary image of a radio over a serial line and twiddle bits before uploading it again.\u00a0 Since the radios are embedded microprocessors with limited storage, lots of bit packing is performed to be as efficient as possible, which results in a lot of bit whacking on my part to pack and unpack the information I need.\u00a0 So far, this has been done by combining things like the struct<\/span> module with the bitwise operators available in Python.\u00a0 That means that the code looks like this:<\/p>\n

\n
tsval = (ord(mmap[POS_TSTEP]) >> 4) & 0x0F\r\nfval = 'x00' + mmap[POS_FREQ_START\"Smilie:OS_FREQ_END]\r\nfreq = ((struct.unpack(\">i\", fval)[0] * mult) \/ 1000.0)\r\nval = struct.unpack(\"B\", mmap[POS_DUPX])[0] & 0xC0\r\nif val == 0xC0:\r\n  duplex = \"+\"\r\nelif val == 0x80:\r\n  duplex = \"-\"\r\nelse:\r\n  duplex = \"\"<\/pre>\n<\/blockquote>\n
It’s really quite ugly, and the bigger issue is that all the code above requires similar but opposite code to poke new values back into the locations.\u00a0 This often resulted in code that would extract the values properly, but not insert them back correctly.\u00a0 Further, common tasks were often done differently in different radio drivers that were written at different times.\u00a0 An example of such a task is extracting an index into an array of values that was, say, three bits in the middle of a byte (or worse, six bits across two different bytes).\u00a0 Another one is packing and unpacking BCD frequency values of various widths and endian orientations.\u00a0 In summary, the major problems with this sort of approach are:<\/p>\n
    \n
  1. Really ugly, hard-to-read, and hard-to-maintain code<\/li>\n
  2. Different code paths for getting and setting a particular bitfield<\/li>\n
  3. Multiple different styles and algorithms for getting at commonly-formatted, but differently-arranged bitfields<\/li>\n
  4. Stability and correctness issues stemming from all of the above<\/li>\n<\/ol>\n
    Recently, I was reading some code by Dean AE7Q that was able to read and write ICOM ICF files.\u00a0 His code is written in C++, so it has the benefit of using structs to map fields in a given buffer.\u00a0 However, what really shook me, was the use of bitfield<\/a> definitions.\u00a0 Now, it’s not like I don’t write C all day long, and interact with bitfield definitions on a regular basis, I just handn’t even considered how much easier they would make the process.\u00a0 I decided that there had to be a way I could do something similar in Python to make my life easier.<\/p>\n
    What I decided to do was write a parser for a simple meta language that looks like C.\u00a0 It needed to support the following elements:<\/p>\n
      \n
    1. Structures<\/li>\n
    2. Arrays<\/li>\n
    3. Bitfield definitions<\/li>\n<\/ol>\n
      Additionally, it would be very helpful if it had the following properties:<\/p>\n
        \n
      1. A native data type for BCD-encoded bytes (many radios store integer values in BCD)<\/li>\n
      2. 24-bit integers (Several radios use a three-byte integer to represent a frequency)<\/li>\n
      3. A few compiler-directives to help seek within a large data stream to a position before mapping<\/li>\n
      4. A pythonic<\/a> mechanism to read and write the defined fields in the data stream after parsing<\/li>\n<\/ol>\n
        What I came up with was a module called bitwise.\u00a0 It uses the PyPEG<\/a>, which is somewhat like lex for C-based compiler writers.\u00a0 The grammar<\/a> is extremely simple and easy to understand, and the parser<\/a>\/compiler\/whatever isn’t too bad either (although it could use quite a bit more cleanup).\u00a0 The result is a simple, single-call interface that turns a binary data stream into a very usable object tree.\u00a0 See the following example code:<\/p>\n
        \n
        # Defines a format for parsing some binary data\r\ndefn = \"\"\"\r\n  struct {\r\n    u8 foo;\r\n    u8 highbit:1,\r\n       sixbits:6,\r\n       lowbit:1;\r\n       char string[3];\r\n       bbcd fourdigits[2];\r\n  } mystruct[1];\"\"\"\r\n# Some binary data for us to parse\r\ndata = \"x7Fx81abcx12x34\"\r\ntree = parse(defn, data)\r\nprint \"Foo: %i\" % tree.mystruct.foo\r\nprint \"Highbit: %i\" % tree.mystruct.highbit\r\nprint \"Sixbits: %i\" % tree.mystruct.sixbits\r\nprint \"Lowbit: %i\" % tree.mystruct.lowbit\r\nprint \"String: %s\" % tree.mystruct.string\r\nprint \"Fourdigits: %i\" % tree.mystruct.fourdigits<\/pre>\n<\/blockquote>\n
        Which prints the following:<\/p>\n
        \n
        Foo: 127\r\nHighbit: 1\r\nSixbits: 0\r\nLowbit: 1\r\nString: abc\r\nFourdigits: 1234<\/pre>\n<\/blockquote>\n
        If I wanted to change the sixbits<\/span> field from all zeros to the value 13<\/span> and the string to “xyz”, all I have to do is:<\/p>\n
        \n
        tree.mystruct.sixbits = 13\r\ntree.mystruct.string = \"xyz\"<\/pre>\n<\/blockquote>\n
        I think that’s a clear win in terms of improved syntax and maintainability.\u00a0 For images where the memory regions I care about are not contiguous from the beginning (which is all of them), the #seek and #seekto directives allow me to apply the definitions that follow to a specific location in the data<\/p>\n
        Aside from the obvious syntactic and mental health gains, here are a few “lines of code” metrics from before and after converting several drivers to use my “bitwise” module:<\/p>\n\n\n\n\n\n\n
        \u00a0Driver<\/strong><\/span><\/td>\nLines before
        \n<\/strong><\/span><\/td>\n        		Lines after
        \n<\/strong><\/span><\/td>\n        		Change
        \n<\/strong><\/span><\/td>\n<\/tr>\n
        \u00a0IC-2820<\/td>\n704<\/td>\n336<\/td>\n53% smaller<\/td>\n<\/tr>\n
        \u00a0ID-880<\/td>\n600<\/td>\n360<\/td>\n40% smaller<\/td>\n<\/tr>\n
        \u00a0VX-8R<\/td>\n375<\/td>\n139<\/td>\n63% smaller<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
        To me, that’s fairly significant, especially given that the code to enable this change is only 721 lines on it’s own.\u00a0 By the way, all of these numbers include comments and a block of (unchanged) license text at the top, so the actual change of functional lines is even more significant.\u00a0 In all, before the change CHIRP had almost 6,000 lines of bit-twiddling driver code.\u00a0 I’ve only converted a few of them so far, but if these initial gains continue to apply, that could cut the total driver codebase by 50%, which is a really<\/strong><\/em> good thing. The other aspect of it, which is harder to convey with numbers, is the fact that I know<\/strong><\/em> that the driver code is more robust now.\u00a0 Since I don’t own all the radios that CHIRP supports, being able to make small maintenance tweaks to the code without having to brute-force test the driver against the real device is extremely helpful.\u00a0 Now that the code is much more symmetric, I think such maintenance tasks will be much smoother in the future.<\/p>\n","protected":false},"excerpt":{"rendered":"
        One of my side-projects is CHIRP, which is an application for programming the memory contents of various radios.\u00a0 Like other projects I’ve done where I want to cater to (or at least, not exclude) people running Windows, it’s written in … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12],"tags":[53,48],"class_list":["post-70","post","type-post","status-publish","format-standard","hentry","category-codemonkeying","tag-chirp","tag-python"],"yoast_head":"\nA better way to process binary data in Python - Right Angles<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"A better way to process binary data in Python - Right Angles\" \/>\n<meta property=\"og:description\" content=\"One of my side-projects is CHIRP, which is an application for programming the memory contents of various radios.\u00a0 Like other projects I’ve done where I want to cater to (or at least, not exclude) people running Windows, it’s written in … Continue reading →\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/\" \/>\n<meta property=\"og:site_name\" content=\"Right Angles\" \/>\n<meta property=\"article:published_time\" content=\"2010-11-10T00:05:49+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2012-02-09T16:11:19+00:00\" \/>\n<meta name=\"author\" content=\"Dan\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Dan\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/\"},\"author\":{\"name\":\"Dan\",\"@id\":\"https:\/\/www.danplanet.com\/blog\/#\/schema\/person\/0f6920aa6d63cae437bf8b122200287c\"},\"headline\":\"A better way to process binary data in Python\",\"datePublished\":\"2010-11-10T00:05:49+00:00\",\"dateModified\":\"2012-02-09T16:11:19+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/\"},\"wordCount\":975,\"publisher\":{\"@id\":\"https:\/\/www.danplanet.com\/blog\/#\/schema\/person\/0f6920aa6d63cae437bf8b122200287c\"},\"keywords\":[\"chirp\",\"python\"],\"articleSection\":[\"Codemonkeying\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/\",\"url\":\"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/\",\"name\":\"A better way to process binary data in Python - Right Angles\",\"isPartOf\":{\"@id\":\"https:\/\/www.danplanet.com\/blog\/#website\"},\"datePublished\":\"2010-11-10T00:05:49+00:00\",\"dateModified\":\"2012-02-09T16:11:19+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.danplanet.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"A better way to process binary data in Python\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.danplanet.com\/blog\/#website\",\"url\":\"https:\/\/www.danplanet.com\/blog\/\",\"name\":\"Right Angles\",\"description\":\"If they're not right...they're wrong\",\"publisher\":{\"@id\":\"https:\/\/www.danplanet.com\/blog\/#\/schema\/person\/0f6920aa6d63cae437bf8b122200287c\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.danplanet.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/www.danplanet.com\/blog\/#\/schema\/person\/0f6920aa6d63cae437bf8b122200287c\",\"name\":\"Dan\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.danplanet.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9b73782704be64dd8c030087af2d1ae0c1dc488cad69093ff0366dbaad2de673?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/9b73782704be64dd8c030087af2d1ae0c1dc488cad69093ff0366dbaad2de673?s=96&d=mm&r=g\",\"caption\":\"Dan\"},\"logo\":{\"@id\":\"https:\/\/www.danplanet.com\/blog\/#\/schema\/person\/image\/\"},\"url\":\"https:\/\/www.danplanet.com\/blog\/author\/dan\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"A better way to process binary data in Python - Right Angles","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/","og_locale":"en_US","og_type":"article","og_title":"A better way to process binary data in Python - Right Angles","og_description":"One of my side-projects is CHIRP, which is an application for programming the memory contents of various radios.\u00a0 Like other projects I’ve done where I want to cater to (or at least, not exclude) people running Windows, it’s written in … Continue reading →","og_url":"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/","og_site_name":"Right Angles","article_published_time":"2010-11-10T00:05:49+00:00","article_modified_time":"2012-02-09T16:11:19+00:00","author":"Dan","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Dan","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/#article","isPartOf":{"@id":"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/"},"author":{"name":"Dan","@id":"https:\/\/www.danplanet.com\/blog\/#\/schema\/person\/0f6920aa6d63cae437bf8b122200287c"},"headline":"A better way to process binary data in Python","datePublished":"2010-11-10T00:05:49+00:00","dateModified":"2012-02-09T16:11:19+00:00","mainEntityOfPage":{"@id":"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/"},"wordCount":975,"publisher":{"@id":"https:\/\/www.danplanet.com\/blog\/#\/schema\/person\/0f6920aa6d63cae437bf8b122200287c"},"keywords":["chirp","python"],"articleSection":["Codemonkeying"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/","url":"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/","name":"A better way to process binary data in Python - Right Angles","isPartOf":{"@id":"https:\/\/www.danplanet.com\/blog\/#website"},"datePublished":"2010-11-10T00:05:49+00:00","dateModified":"2012-02-09T16:11:19+00:00","breadcrumb":{"@id":"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.danplanet.com\/blog\/2010\/11\/10\/a-better-way-to-process-binary-data-in-python\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.danplanet.com\/blog\/"},{"@type":"ListItem","position":2,"name":"A better way to process binary data in Python"}]},{"@type":"WebSite","@id":"https:\/\/www.danplanet.com\/blog\/#website","url":"https:\/\/www.danplanet.com\/blog\/","name":"Right Angles","description":"If they're not right...they're wrong","publisher":{"@id":"https:\/\/www.danplanet.com\/blog\/#\/schema\/person\/0f6920aa6d63cae437bf8b122200287c"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.danplanet.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/www.danplanet.com\/blog\/#\/schema\/person\/0f6920aa6d63cae437bf8b122200287c","name":"Dan","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.danplanet.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/9b73782704be64dd8c030087af2d1ae0c1dc488cad69093ff0366dbaad2de673?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9b73782704be64dd8c030087af2d1ae0c1dc488cad69093ff0366dbaad2de673?s=96&d=mm&r=g","caption":"Dan"},"logo":{"@id":"https:\/\/www.danplanet.com\/blog\/#\/schema\/person\/image\/"},"url":"https:\/\/www.danplanet.com\/blog\/author\/dan\/"}]}},"_links":{"self":[{"href":"https:\/\/www.danplanet.com\/blog\/wp-json\/wp\/v2\/posts\/70","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.danplanet.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.danplanet.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.danplanet.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.danplanet.com\/blog\/wp-json\/wp\/v2\/comments?post=70"}],"version-history":[{"count":4,"href":"https:\/\/www.danplanet.com\/blog\/wp-json\/wp\/v2\/posts\/70\/revisions"}],"predecessor-version":[{"id":217,"href":"https:\/\/www.danplanet.com\/blog\/wp-json\/wp\/v2\/posts\/70\/revisions\/217"}],"wp:attachment":[{"href":"https:\/\/www.danplanet.com\/blog\/wp-json\/wp\/v2\/media?parent=70"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.danplanet.com\/blog\/wp-json\/wp\/v2\/categories?post=70"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.danplanet.com\/blog\/wp-json\/wp\/v2\/tags?post=70"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}