“You can make it painful” or “PyPI + lack_of_knowledge = kill_yourself”
Posted by: mishunov on: May 29, 2009
- In: Development | Plone
- 2 Comments
- So, what is it about?
- It is about how painful things can be when some bugs overlap each other in the same process.
- … excuse me, but really, what is it about?
Everything has started when I released the version 2.0 of webcouturier.dropdownmenu. While making that release, I decided to add a better description for the package. The idea was good, the release worked as it should and I left for my vacation.
But during the vacation I got tons of emails from people complaining that this package breaks their buildouts – buildout can’t get webcouturier.dropdownmenu package and thus can’t finish the building process. Some have reported that the problem is that buildout tries to connect to http://www.webcouturier.com and that attempt fails. I would like to thank all people who have spent their time to send me tracebacks and information about the problems they have.
Lyrical digression about http://www.webcouturier.com. This is the project that I have been running in 2006-2008. Some might remember my slides from The Plone Conference in Naples in 2007 where I have used the screenshots of that project. These days (after I started to work with Jarn in Norway) this project doesn’t exist in it’s previous form. Though the project might be back at some point in a slightly transformed way. Anyway the site is down for a couple of months now.
So, right after my vacation I decided to solve the issue as soon as possible. I tried to understand where that URL comes from. The only place where the URL is mentioned, is within the packages README.txt file. After one evening without a result, I have discussed this with my colleagues. Florian Schulze came up with the interesting URL – http://pypi.python.org/simple/webcouturier.dropdownmenu that shows what links are stored for the package in PyPI. And setuptools tries to check all of those links when you add a package to your buildout. I have been surprised to find http://www.webcouturier.com as one of the links there. Moreover all the links to the home pages of each release have been there and they were all linked to http://www.webcouturier.com as well. Well, well, well… And now the question is – even if I define any links for my package (home page in setup.py for example) why would setuptools want to forward my buildout to the link that doesn’t exist? Wouldn’t it be nice to make anything like links’ checkup on either setuptools or PyPI level that would remove dead links from the index list? Anyway, the source of the problem has been identified and it was the way setuptools looks for releases combined with the way PyPI creates the simple index.
But PyPI can not add links from nothing. That means there was something in my package that adds them. First of all I have decided to simply get rid of “Home Page” link for all releases. This got rid of those “home page” links linked to webcouturier.com in the package index links list. After that, there has been only one entry for webcouturier.com left that was not obvious where comes from. I have tried to search and the only place that popped up was that README.txt file. The content of that file is used in setup.py as a long description for a package. Plus the same information is used as a package’s description on PyPI. “Package’s description definitely should not be parsed for package’s index links,” I thought, “information in description is definitely not the place to search for important links to get the package from.” But as long as description was the only place mentioning that URL I have decided to try and manually remove the links from description on PyPI.
You wont believe, but that was it – the dead link from the package’s index links was gone after that. Whow! I was confused. How can one guess that in case you have a dead link in, let’s say, your very first release, it will break your release #100 for example? And how can one avoid using URLs in packages’ descriptions? This sounds completely off. Because one can never be sure that the site he gives credit to in one of the releases will always be up and running. Thus there is always a risk that in case that site is down your package breaks buildout. Sounds cool, doesn’t it?
As a conclusion
I have no clue what idea is behind such behavior if any. I can even imagine that developers (that I am technically not) might understand that idea. But as a UI person I was absolutely confused with how things work and how simple it is to break a buildout for thousands of sites without even intentional wish. From my point of view there are two main problems that have to be either explained somewhere or fixed (since these are bugs in my understanding):
- Dead links should not be parsed by setuptools when running a buildout. Ideally they should be cut off even before getting to index links list. I am not sure how this should be achieved technically but it has to at some point.
- Links in a package’s description should not be parsed or at least should be marked somehow in a manner that buildout would not try to reach the resources behind those links.
- setuptools have to ignore duplicated URLs in a package’s index. If there are links, linked to the same resource what’s the point of getting that resource more than once? It’s just a waste of time and resources.
As a bonus
As a bonus I would like to let readers know that the problems like this can be solved on a buildout level for every buildout separately. Means you don’t need to ping author of the package and wait while he fixes it – you can fix it (at least to have a temporary solution) locally. Check the nice package by Florian Schulze, called jarn.setuptoolsfixer. Simply add one more line to your buildout.cfg within [buildout] section:
[buildout]
...
extensions = jarn.setuptoolsfixer
and that’s it. What this package does is “fixes a bug in setuptools which makes it fail when the homepage or download url on a PyPI page is broken. If the egg is stored on PyPI then this fix allows buildout to continue”. So, this can be useful when you are getting Download error: (60, 'Operation timed out') -- Some packages may not be found! error and your buildout is broken due to this. Just note that the buildout may run longer than usual, because broken links need to time out. In case of webcouturier.dropdownmenu setuptools tried the links of all the releases, even though they point to the same URL, which meant it had to time out for roughly 8 links and the default timeout is 60 seconds!
Long live your buildouts and Plone projects 
Good luck!
Leopard, Python and other wild creatures
Posted by: mishunov on: October 24, 2008
- In: Development | Plone
- 2 Comments
The post is mostly a reminder to myself and contains references to some other posts by other respected developers.
Today I have found out that Philipp von Weitershausen has written the post about the same problem I had a couple of weeks ago and was in the middle of writing the post about the same. Philipp has already explained the problem but I would like to extend it a little (since it’s the note to myself as well):
- The problem for me showed up not only with Python 2.5, but also with Python 2.4. Since Plone is still using Python 2.4 this is a real problem for me
- There is no problem when you are using Mac OS X Tiger. Only Leopard
- There is no problem when you have installed Leopard by updating the Tiger. Problem only strikes clean installations of Leopard
- Bus Error shows up only in the interpreter shell, but interpreter itself can work well. Means you can not notice the problem if you just run Plone and don’t use Python’s shell as with PDB (not this blog, but python debugger) for example
So, in general, when you install Python in Leopard using MacPorts you are doomed to have Buss Error, since python will pick up broken readline from the system. As Steve McMahon points out in Philipp’s post and Andreas Jung in his own post Leopard uses broken libedit or libreadline (I still don’t get what Leopard is actually using). And that broken library will be caught by your python when you install it with MacPorts.
To avoid the problem I had to do the following steps. They worked for me pretty fine and might work the same nice for you as well. Though it’s not guarantied I also have to point out that I needed Python 2.4, so there might be changes for other versions.
Install new readline
It’s safe to install it with MacPorts:
sudo port install readline
Install Python from source
I have pointed out the problem with readline when you install python with MacPorts on Leopard. So I have decided to build python from source. If you will simply try to download python 2.4 and configure it you can get something like:
lector:Python-2.4.5 denys$ ./configure --prefix=/Users/denys/opt
...
lector:Python-2.4.5 denys$ make
...
gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE - ./Modules/posixmodule.c -o Modules/posixmodule.o
./Modules/posixmodule.c: In function 'posix_setpgrp':
./Modules/posixmodule.c:3145: error: too few arguments to function 'setpgrp'
make: *** [Modules/posixmodule.o] Error 1
If you have this, this means you are really using Leopard. And Python 2.4 really can’t be installed in Leopard out-of-the-box. Bah! Even if it sounds silly what you need to do is to explicitly tell python to be installed for Leopard
./configure MACOSX_DEPLOYMENT_TARGET=10.5
But remember we need to point our python to the newly installed readline to avoid Buss Error.
According to Andreas Jung right after configuring your python and before running make you should roughly invade your Makefile. I have no wish to discuss how dirty this solution is, but it works. So, what needs to be done is replacing CC = gcc line in your Makefile to
CSS = gcc -I/opt/local/include -L/opt/local/lib
Then we can finally do
make && make install
If everything went smooth you can try simple test
lector:/ denys$ python2.4
Python 2.4.5 (#1, Oct 31 2008, 22:28:08)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 2+2
4
>>> 2+3
5
Previously (using python installed with MacPorts) you would get Buss Error after “2+3″ request. That’s it about installing python from source in Leopard.
If you plan to use this version of python to run Plone instances, don’t forget to make it your default by putting path to your python in your ~/.profile. Something like
export PATH=/Users/denys/opt/bin:$PATH
Install additional libraries
Of course you would like to have more libraries.
- easy_install. Download and run
lector:Imaging-1.1.6 denys$ python ez_setup.py
- PIL. I prefer to install it from source with
lector:Imaging-1.1.6 denys$ python setup.py build_ext -i
...
lector:Imaging-1.1.6 denys$ python setup.py install
- ZopeSkel
lector:Imaging-1.1.6 denys$ easy_install ZopeSkel
- pyflakes. Really good checker for simple bugs (typos, wrong indentation, wrong variables etc.) in python.Previously it was possible to install it with easy_install. But these days author does not care a lot about updating product’s page on pypi. So the only one way of installing it is from source. So
lector:Downloads denys$ svn co http://divmod.org/svn/Divmod/trunk/Pyflakes
lector:Downloads denys$ cd Pyflakes/
lector:Pyflakes denys$ python setup.py install
In general, you should have PyFlakes installed at this point. At least I would expect it to. But, if you will try to run pyflakes in your terminal, you will most probably get something like:
...
Traceback (most recent call last):
File "/Users/denys/opt/bin/pyflakes", line 3, in ?
from pyflakes.scripts.pyflakes import main
ImportError: No module named scripts.pyflakes
I have no idea why, but the installation procedure of the Pyflakes does not copy scripts/ folder to your python’s site-packages/pyflakes. So, this is the step you need to do manually – copy scripts/ from your SVN checkout of Pyflakes to your site-packages/pyflakes. In my case it was
lector:Pyflakes denys$ cp -r ~/Downloads/Pyflakes/pyflakes/scripts/ ~/opt/lib/python2.4/site-packages/pyflakes/scripts/
Now your Pyflakes should work properly.
Hello world or why “Pure Drivel”
Posted by: mishunov on: October 22, 2008
drivel
noun [U] DISAPPROVING
nonsense or boring and useless information
Cambridge Online Dictionary
So, why does this blog called “Pure Drivel Blog”?
First – majority of blogs are the examples of drivel by default. So, why this one should be different?
Second – it is some how influenced by the audiobook I have listened to recently – “Pure Drivel” by Steve Martin. That book made me think… (some people, who have read this book, might laugh about the fact of “thinking” while reading/listening to that book) …that a huge number of things in our lives are just the drivel and people should not take everything seriously.
So, just don’t take everything that is written here seriously. Some things just don’t suppose to be serious
Denys' Photos on Flickr
|
