Sunday, January 27, 2013

Slide sharing is hard

As the result of the previous post, I have discovered that sharing presentations is hard and inconvenient. Perhaps this is just for Keynote presentations. When I was trying to share those slides, I wanted to:

  1. include my presenter's notes, since the slides need context.
  2. display animations, since I wasted a lot of time trying to make sure that the correct things animated during transitions in Keynote. I think animations do well at highlighting changes between slides, and in this particular case, I think it helps me illustrate control flow and concurrency better.

As it turns out, only one of these is easy to do.

An incomplete survey of Keynote slide sharing options:
  • Export to PDF: The easiest way to distribute slides is obviously to export them to PDF, since almost everyone can view PDFs. Keynote can even export slides with presenter's notes, which is what I did. I should have just stopped here, but PDFs don't have animations or slide transitions.
  • Exporting to HTML: This seems to allow transitions to display, at least theoretically, in Safari (as it turns out, it didn't seem to work for me). But Keynote does not export presenter's notes along with the slides - so I wrote a gross hack to extract the presenter notes and stick them into the exported HTML sldies.
  • Exporting as Quicktime Movie: Again, this fails to export presenter notes. Perhaps I can hack something together with presenter notes in the subtitles, but generally I don't like watching slides or how-tos in video - I'd much rather read text.
  • Uploading to SlideShare, which accepts slides in various formats. It will also display a transcript of the slides for accessibility, and extract speaker notes and display those next to the slides. I also liked that it lets you embed your presentation, rather than just provide a link to a presentation file. Several problems though:
    • Supposedly SlideShare supports Keynote '09 now, but technical problems seem to have caused them to retract said support. So I had to export to some other format anyway.
    • Exporting to PPT format and then uploading to SlideShare results in no presenter notes. I'm not sure if this is because Keynote fails to export presenter notes to PPT format or that SlideShare cannot pull presenter notes out of PPT.
    • Uploading the exported PDF slides with presenter notes caused the accessibility transcript to contain both the text in my slides and text in my presenter notes, mashed together.
  • Uploading to Speaker Deck, which is similar to slideshare in that it lets you embed your slides somewhere (like a blog). It only accepts PDFs and has no accessibility transcripts or slide nodes. It has a nice clean interface. I ultimately went with uploading my exported PDF (with presenter notes) to SpeakerDeck, since it was functionally the same as uploading my slides to SlideShare anyway, except the UI was nicer and it lacks a confusing transcript extract.

If anyone has a better solution to suggest, please let me know.

Saturday, January 26, 2013

Introduction to Twisted presentation

Earlier this month, I gave a brief Introduction to Twisted presentation to other developers at Rackspace, San Antonio, as a part of Rackspace's internal Mindshare program.

This is heavily cribbed from Jessica McKellar's Architecture of Open Source section on Twisted and the Krondo tutorials.

Hopefully these slides might be helpful to other people: (and are also available as a PDF download outside of SpeakerDeck as well as an HTML presentation).

Sunday, July 15, 2012

Twisted on Windows using ActiveState Python

We recently held a Learn Twisted event at the SF Twisted Meetup, and craig (one of the attendees) linked a fantastic writeup of how to set up Twisted on Windows using ActiveState Python. He kindly gave me permission to use it this blog post.

Installation Steps

  1. Install Windows 32-bit version of ActiveState Python from .
  2. Open a Windows Command Prompt, and run this command:
        pypm install twisted
  3. Verify that you have installed Twisted by running this command:
        pypm ls


  1. The Windows 32-bit ActiveState Python version must be installed, not the Windows 64-bit ActiveState Python version. If you install the Windows 64-bit version,otherwise you will need to purchase a Business Edition license from ActiveState, in order to install some Python modules which are "locked".
  2. Many other Python modules can be installed via pypm. See:
    More documentation about pypm can be obtained by typing:
       pypm help
  3. This step is completely optional, and not necessary for using Twisted for the class exercises.

    If you ever want to build Python modules which have native code in C, you will probably need to have the Microsoft Visual C++ compiler installed. It is best to get a license and install the full version of Visual C++.

    If you don't have the full version of Visual C++, you can try installing the free Visual C++ Express edition from Microsoft.

Monday, March 12, 2012

Twisted Development on Windows (v2)

At the first Sprint night at PyCon 2012, we ran into some trouble setting up Twisted in Windows. Since it has been a very long time since my original notes on how to develop Twisted in Windows, I thought I'd post an updated version.

This how-to is a work-in-progress, and is directed toward newer Python programmers who would like to work on Twisted (hopefully at the next few sprint days!), but I assume the following:

  1. You know how to download files and install programs in Windows.
  2. That Python 2.7 is already installed. If not, the Boston Python User's group has a great wiki page on how to do so.
  3. That you are at least somewhat familiar with using command line, editing and running Python files, etc. If not, again the Boston Python user's group is a great resource.

I did this in Windows XP, but it should be applicable to Windows Vista or Windows 7 as well.

Useful tools you may want before you start this.

These are just the tools that I used, so they will be what you will see in the screenshots, but you can use any alternatives you wish:

  • 7-zip - Useful for decompressing bz2 and gzipped files tar files. Note: if you see a file like "file.tar.gz" or "file.tar.bz2", that means that it is "archived" twice - tar is a container for the files, and gzip and bzip2 compresses that tar container. Just wanted to clarify in case you unzip a "file.tar.gz" and get a "file.tar".

  • Console2 - a nicer command prompt for windows - there doesn't seem to be an installer, just an executable in a zipped folder.

  • Sublime Text 2 - my current preferred text editor (the Boston Python Workshop recommends Notepad++)


  • Install MinGW, which is a GCC compiler for Windows. Download the mingw-get-inst installer. If it asks you for options, you may only need C, and C++, but I just installed everything, including Fortran, and ObjC.

  • Install pywin32, which are the Windows extensions for Python. Downloadable files are under the tab "Files". Go to the latest build and download the build for your version of windows, for Python 2.7. Install the executable even if you have/are planning on installing virtualenv. I haven't figured out how to install it in a specific virtualenv, and I haven't been able to google up an answer.

  • Install setuptools (which includes easy_install), which lets you install python extensions via running 'python' on source packages. Download the latest version for Windows that for Python 2.7 (download the .exe at the bottom of that page) and install it.

Optional, but recommended

Install pip, which manages python packages - it makes getting/installing/removing python packages later a lot easier. These instructions are mostly snarfed from this stackoverflow question):

  1. Download the source (close to the bottom of the page) and uncompress it anywhere (the uncompressed files can be deleted after it is installed)

  2. On a console, go to the uncompressed pip directory and run install pip:
       cd [pip directory]
    python install
  3. (if you haven't already added 'c:\Python27\Scripts' to your PATH environment variable (see the Windows Python setup wiki), do so.

To verify that everything installed correctly, try typing:

   pip --version
at the command line. If you don't get an error, then pip has been installed correctly. (In the screenshot, the 'SNIP' just means there was a lot more text that I cut out so as to fit the relevant bits in an image of reasonable size).

Entirely Optional

Virtualenv is something that creates isolated python environments. For instance if you want to work with different versions of packages such as Twisted or Django, it lets you set up environments that are independent of each other, so you don't have to edit your Python path to switch between versions, or do any other hacks. As far as I can tell it works well enough in windows, although some options are not available.

If you haven't already installed pip, do so first.

  1. Install virtualenv by typing at the command console:
       pip install virtualenv
  2. Set up a twisted virtualenv (virtual environment in which twisted will be run):
    1. Go to C:\ (or wherever you wish to keep your virtual environments), and create a new virtual environment by typing this at the command line:
         virtualenv --system-site-packages TwistedEnv
      (You don't have to name your virtual environment "TwistedDev") Do not forget the "--system-site-packages" option, because otherwise pywin32 will not be installed in this virtual environment.
    2. This creates a directory (a folder) named "TwistedDev". Go into that directory activate the environment by typing at the command line:
         cd TwistedEnv\Scripts
    3. Notice that the prompt now says (TwistedEnv) before the C:\. This means that you are now working in the TwistedEnv virtual environment. Once you are in this environment, if you use "pip install" or "python install", the python packages will be installed to the TwistedEnv\Lib\site-packages directory, rather than the C:\Python27\Lib\site-packages directory.

Get and install Twisted

  1. Get Twisted. Go to and download the source archive (make sure to not to download a windows installer). For more advanced users: check out Twisted from SVN instead.

    Alternately, if you do not want to actually work on Twisted but just use it, you can do the following (and you can probably skip the rest of the instructions):

       easy_install twisted

  2. Save and extract it to C:\TwistedEnv\ (or wherever your twisted virtual environment is). If you did not/do not want to install virtualenv, then just extract it wherever (e.g. C:\).

  3. "Install" it by typing at the command line (this causes an error, but the solution is after this explanation):

       python develop
    Usually the way to install it is 'python install', but that just copies the relevant files to your site-packages directory. You want to actually work on Twisted itself though, so using 'develop' basically just creates a link in site-packages to your source directory (C:\TwistedEnv\Twisted-12.0.0, or C:\Twisted-12.0.0 if you do not have virtualenv).

    You will probably see some vcvarsall.bat errors. This is because distutils\ uses a -mno-cygwin, which has been removed from the latest version of mingcw32.

  4. Edit twisted's so that the getExtension function returns an empty list:

    Then re-run
       python develop

    There is possibly a better way, because it will let you build other python extensions with MinGW later, but I haven't gotten it to work. If anyone has a solution please let me know. As per this stackoverflow answer, you can edit C:\Python27\Libs\distutils\ In the class Mingw32Compiler, change the __init__ method to remove the '-mno-cygwin' option. That didn't seem to fix the vcvarsall.bat error for me. I also tried to add setup.cfg file to distutils with the following:


  5. Make sure it installed correctly. Back out of your current working directory (the Twisted source directory) into your TwistedEnv directory (alternately, to C:\ if you did not install virtualenv), and try importing twisted.

       cd ..
    At the prompt, type:
      >>>import twisted
    >>>print twisted
    If twisted installed correctly, importing the twisted module will not raise any exceptions. Also note that the twisted module that is loaded is the one in your source directory.

    Twisted depends on zope.interface (says so on the Twisted website). If you scroll up in your console, you'll see that that downloaded and installed it for you. (see above screenshot)


No seriously, this post is somewhat unfinished. I haven't explained how you'd work on different Twisted branches after running 'python develop', for instance, since I do not know the recommended way to do so. Someone at some point will probably write up something.

Sunday, October 30, 2011

Twisted development using Sublime Text 2

Been experimenting with Sublime Text 2 recently instead of vim. I installed SublimeLinter - it's pretty cool! It uses pyflakes and pep8 to verify your Python source formatting:
(when mouse-ed over the unused import)
However, the blank line requirements according to PEP 8 are different from those of the Twisted Coding Standard. So I monkey-patched pep8 (you can just dump the monkey-patching script into the Sublime Text Packages/User directory). So 3 blank lines between class definitions, and 2 between class method definitions:
(when mouse-ed over the class definition, which does not have enough preceding blank lines)

Friday, October 2, 2009

Amazon's Whispernet: the door swings both ways

Amazon's Whispernet and Whispersync have been touted as a revolution, an innovation in the content pipeline that will propel eBooks from a niche market into the mainstream population. If you have a Kindle, Whispernet means that you have access to all of Amazon's Kindle content from anywhere there is Sprint coverage, and that your user-generated content is automatically backed-up and synced across multiple devices by Amazon. For free. Great! It also means that Amazon pulls data from your Kindle; obviously, Amazon retrieves the content that it backs up, but perhaps it gets other information as well. To co-opt an old adage: if you can see Amazon, Amazon can see you.  And what Amazon sees (and how often) affects your data privacy and ownership.  Given Amazon's track record with transparency, Kindle users should be proactive in learning more about features of their device that are not advertised.

So just what information is sent to Amazon by your Kindle? According to Amazon:
"Information Received. The Device Software will provide Amazon with data about your Device and its interaction with the Service (such as available memory, up-time, log files and signal strength) and information related to the content on your Device and your use of it (such as automatic bookmarking of the last page read and content deletions from the Device). Annotations, bookmarks, notes, highlights, or similar markings you make in your Device are backed up through the Service. Information we receive is subject to the Privacy Notice."
This description is very vague. Just what kind of information is logged on the device, and how much of it is sent to Amazon? What information related to content is sent? Does Amazon only receive information related to content that is downloaded from Amazon, or will they receive information about, for instance, the fan-fiction you downloaded? How often is this information sent to Amazon?

If the Kindle communicated with Amazon via WiFi, you could set up your access point or router to intercept traffic from the Kindle and find the answers to the above questions that way. However, the Kindle communicates with Amazon over a Sprint 3G network via an account you have no access to, so you'd have to somehow intercept and interpret the cell signal. My understanding of wireless technology is extremely limited, but I'm guessing that you'd have to either passively intercept the signal (a federal crime in the United States) or somehow execute a man-in-the-middle attack against your Kindle (for instance, by setting up or impersonating a cell station, which would also probably constitute at least one federal crime).

Without being able to intercept traffic from the Kindle, the only way to determine exactly what information is sent by a Kindle is hacking it, which is against the Amazon Kindle License and Terms of Use:
No Reverse Engineering, Decompilation, Disassembly or Circumvention. You may not, and you will not encourage, assist or authorize any other person to, modify, reverse engineer, decompile or disassemble the Device or the Software, whether in whole or in part, create any derivative works from or of the Software, or bypass, modify, defeat or tamper with or circumvent any of the functions or protections of the Device or Software or any mechanisms operatively linked to the Software, including, but not limited to, augmenting or substituting any digital rights management functionality of the Device or Software.

Luckily, there are people far more tech-savvy, more willing to risk bricking their Kindles, and more willing to risk getting banned by Amazon than I am. They've who have figured out what information gets sent to Amazon. Since the sample logs from that forum post are just snippets, my interpretation may not be entirely accurate, but what seems to get sent to Amazon is:
  • the times at which you switch screens (i.e. from the list of books to a particular book)
  • the details of the book you are reading (i.e. the title, authors, the Amazon Standard Identification Number, content type, publisher, publication date, display title and authors, length, when you last accessed the book, your last location in the book, whether it's encrypted, whether it's a sample, whether it is newly downloaded to your Kindle, the path to the file on the Kindle system, whether there is text-to-speech metadata)
  • the details of your device - I'm not sure how to interpret all the data, but it seems like: your EVDO network information, signal strength, your latitude and longitude, and more
Obviously, because the Kindle has a 3G radio, Sprint knows where you are. But why is this information sent to Amazon as well? The terms of use did not mention receiving information regarding your location coordinates.

From the various complaints I've read about Whispersync not applying to PDFs on the Kindle DX, and the phrasing in various reviews regarding Whispersync, I assume Amazon does not sync your non-Amazon content across your devices. However, it's possible that Amazon also retrieves statistics about your non-Amazon content, along with your Amazon content (i.e. title, filename, etc. as given above).

Furthermore, when you use the web browser, all traffic goes through an Amazon web proxy (which is understandable - the proxy can optimize pages for display on the Kindle, for example by filtering out large images and/or video that the Kindle can't display anyway). This information is not disclosed to the Kindle user (although perhaps Amazon did not mention this in the Kindle terms of use because the browser is experimental). So if you use the Kindle for web browsing, Amazon also receives information about which sites you visit.

There is an obvious reason why Amazon wants all this data, a reason that doesn't involve tinfoil hats: market research. The wealth of information regarding users' reading habits provides accurate data I doubt companies can even pay for because self-reporting is notoriously subject to recall bias. Setting up a market research study at such a scale, with a sample size numbering in the hundreds of thousands, if not millions, would probably be prohibitively expensive. The data the Kindle collects is commercially valuable to Amazon, possibly valuable enough to offset what it pays to Sprint for connectivity.

While Amazon's motivations are likely benign, there may still be negative externalities associated with the data collection.  For instance, although this information is generated by their users, the users cannot access it and have no control over how long it is stored or how it is used.  Is losing control over personal data a reasonable price for a user to pay for the convenience of Whispernet? Maybe so - it depends on the preferences of each Kindle user.  However, the price is likely one that most users aren't even aware of and do not know how to evaluate. Which brings me to another cliché I was considering as a title for this blog post: there is no such thing as a free lunch.