Nevow doesn't declare its dependency on Twisted in a machine-parseable way

I open ticket #2630 (installation of Nevow doesn't have automated tests).

Ah, there are missing newlines in the two minimal projects.

Here they are with newlines:

One that yields the bizarre stack trace about PIL:

import setuptools
setuptools.setup(name="minproj_that_needs_nevow", setup_requires=["Twisted"], install_requires=["Nevow"]

One that yields the nice exception message about Twisted not being installed:

import setuptools
setuptools.setup(name="minproj_that_needs_nevow", install_requires=["Nevow"]

Installing the latter setup.py has the same effect as running easy_install Twisted.

(Note that for my purposes for allmydata.org Tahoe, a nice Python exception stack trace with ImportError: No module named twisted.python is not good enough. I want users who know nothing about Python to install and run Tahoe successfully on their first attempt without ever learning that this required the installation of Nevow nor that the installation of Nevow required the installation of Twisted. I'm 90% of the way there already...)

Here is the ticket on the allmydata.org trac to track this issue as it impacts allmydata.org Tahoe:

http://allmydata.org/trac/tahoe/ticket/440

Waiting for #2657

Maybe you meant waiting for #2630 ? In any case, #2657 and #2630 are both fixed, so shall I submit a patch which causes Nevow to declare its dependency on Twisted in the same way that Twisted declares its dependency on zope.interface?

Oh, I bet that you meant #2527. #2527 is not done (and now that I noticed it I will see if I can help). We could do this one without doing #2527, by saying

if 'setuptools' in sys.modules:

The way Twisted does:

http://twistedmatrix.com/trac/browser/trunk/setup.py?rev=23010#L75

But I would be just as happy to do #2527 first and then revisit this one.

Okay now that #2630 is done (except see comments that I added to that ticket after it was closed) and now that #2527 seems to be done (judging by my local trial runs -- not committed to trunk or buildbot-tested yet), here is a patch that would close this ticket:

HACL yukyuk:~/playground/nevow/nevow-install-2527/Nevow$ svn diff Index: setup.py =================================================================== --- setup.py (revision 16601) +++ setup.py (working copy) @@ -70,6 +70,8 @@

from setuptools import setup, find_packages

setupdictpackages? = find_packages()

+ + setupdictinstall_requires? = Twisted >= 2.4?

else:

# No setuptools -- decide where the data files should go and explicitly list # the packages.

Whoops, here it is again quoted against traciwikification:

HACL yukyuk:~/playground/nevow/nevow-install-2527/Nevow$ svn diff 
Index: setup.py
===================================================================
--- setup.py    (revision 16601)
+++ setup.py    (working copy)
@@ -70,6 +70,8 @@
     from setuptools import setup, find_packages
 
     setupdict['packages'] = find_packages()
+
+    setupdict['install_requires'] = ["Twisted >= 2.4"]
 else:
     # No setuptools -- decide where the data files should go and explicitly list
     # the packages.

I submitted a patch which hopefully fixes #2527 (easy_install compatibility). If it works, then please apply this patch which hopefully fixes this ticket.

It seems like #2527 is finished now (I'm just waiting for someone to commit it to trunk). So please apply this patch as well.

Please apply this patch, now that #2527 has been merged to trunk:

HACL yukyuk:~/playground/nevow/Nevow$ svn up
At revision 17075.
HACL yukyuk:~/playground/nevow/Nevow$ svn diff
Index: setup.py
===================================================================
--- setup.py    (revision 17075)
+++ setup.py    (working copy)
@@ -81,6 +81,7 @@
     setupdict['packages'] = find_packages()
     setupdict['packages'].append("twisted.plugins")
     setupdict['include_package_data'] = True
+    setupdict['install_requires'] = ["Twisted >= 2.4"]
 else:
     # No setuptools -- decide where the data files should go and explicitly list
     # the packages.

I want to set up a builder which tests this. I think that would mean having a stub "project" which has one of the setup.py scripts from the earlier comments and then easy_installing it and... running the Nevow test suite, I guess?

Yes. Here is a minimal setup.py:

import setuptools
setuptools.setup(name="minproj_that_needs_nevow", install_requires=["Nevow"]

You should be able to do the same install process as the "nevowinstall" builder (http://buildbot.divmod.org/builders/linux32-py2.5-nevowinstall) to install that minproj into a subdirectory. Then add that directory to the PYTHONPATH and ... maybe just import nevow?

I guess we're waiting for someone to write a BuildStep? to test this.

I was unable to reproduce the error from the ticket description:

exarkun@charm:/tmp/nevowinstalltest$ cat setup.py 
import setuptools
setuptools.setup(name="minproj_that_needs_nevow", install_requires=["Nevow"])

exarkun@charm:/tmp/nevowinstalltest$ python setup.py install --root /tmp/nevowinstalltest/foo --single-version-externally-managed 
running install
running build
running install_egg_info
running egg_info
creating minproj_that_needs_nevow.egg-info
writing requirements to minproj_that_needs_nevow.egg-info/requires.txt
writing minproj_that_needs_nevow.egg-info/PKG-INFO
writing top-level names to minproj_that_needs_nevow.egg-info/top_level.txt
writing dependency_links to minproj_that_needs_nevow.egg-info/dependency_links.txt
writing manifest file 'minproj_that_needs_nevow.egg-info/SOURCES.txt'
reading manifest file 'minproj_that_needs_nevow.egg-info/SOURCES.txt'
writing manifest file 'minproj_that_needs_nevow.egg-info/SOURCES.txt'
Copying minproj_that_needs_nevow.egg-info to /tmp/nevowinstalltest/foo/usr/lib/python2.5/site-packages/minproj_that_needs_nevow-0.0.0-py2.5.egg-info
running install_scripts
exarkun@charm:/tmp/nevowinstalltest$ ls -R
.:
foo  minproj_that_needs_nevow.egg-info  setup.py

./foo:
usr

./foo/usr:
lib

./foo/usr/lib:
python2.5

./foo/usr/lib/python2.5:
site-packages

./foo/usr/lib/python2.5/site-packages:
minproj_that_needs_nevow-0.0.0-py2.5.egg-info

./foo/usr/lib/python2.5/site-packages/minproj_that_needs_nevow-0.0.0-py2.5.egg-info:
dependency_links.txt  PKG-INFO  requires.txt  SOURCES.txt  top_level.txt

./minproj_that_needs_nevow.egg-info:
dependency_links.txt  PKG-INFO  requires.txt  SOURCES.txt  top_level.txt
exarkun@charm:/tmp/nevowinstalltest$ 

Should I be invoking setup.py differently? I thought --single-version-externally-managed might be suppressing the behavior where the problem occurs, but I also tried without that and didn't see the problem. If I remote --root then setuptools refuses to try the installation at all, since I don't have write permission to /usr/lib/.

Sorry I didn't notice your comment until now -- I updated #2698 (please mail me when my tickets change) about that.

--root appears to disable the feature of automatically resolving dependencies, so minproj_that_needs_nevow gets installed and Nevow doesn't get installed. Here's how I test these things:

HACK yukyuk:/tmp/nevowinstalltest$ mkdir -p /tmp/nevowinstalltest/foo/lib/python2.6/site-packages ; PYTHONPATH=/tmp/nevowinstalltest/foo/lib/python2.6/site-packages python setup.py install --prefix=/tmp/nevowinstalltest/foo && PYTHONPATH=/tmp/nevowinstalltest/foo/lib/python2.6/site-packages python -c "import nevow;print nevow;print nevow.__version__"
running install
/usr/local/lib/python2.6/dist-packages/setuptools-0.6c7.egg/setuptools/package_index.py:7: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  from md5 import md5
running bdist_egg
running egg_info
writing requirements to minproj_that_needs_nevow.egg-info/requires.txt
writing minproj_that_needs_nevow.egg-info/PKG-INFO
writing top-level names to minproj_that_needs_nevow.egg-info/top_level.txt
writing dependency_links to minproj_that_needs_nevow.egg-info/dependency_links.txt
reading manifest file 'minproj_that_needs_nevow.egg-info/SOURCES.txt'
writing manifest file 'minproj_that_needs_nevow.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
warning: install_lib: 'build/lib.linux-x86_64-2.6' does not exist -- no Python modules to install
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying minproj_that_needs_nevow.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying minproj_that_needs_nevow.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying minproj_that_needs_nevow.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying minproj_that_needs_nevow.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying minproj_that_needs_nevow.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/minproj_that_needs_nevow-0.0.0-py2.6.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing minproj_that_needs_nevow-0.0.0-py2.6.egg
removing '/tmp/nevowinstalltest/foo/lib/python2.6/site-packages/minproj_that_needs_nevow-0.0.0-py2.6.egg' (and everything under it)
creating /tmp/nevowinstalltest/foo/lib/python2.6/site-packages/minproj_that_needs_nevow-0.0.0-py2.6.egg
Extracting minproj_that_needs_nevow-0.0.0-py2.6.egg to /tmp/nevowinstalltest/foo/lib/python2.6/site-packages
minproj-that-needs-nevow 0.0.0 is already the active version in easy-install.pth

Installed /tmp/nevowinstalltest/foo/lib/python2.6/site-packages/minproj_that_needs_nevow-0.0.0-py2.6.egg
Processing dependencies for minproj-that-needs-nevow==0.0.0
Searching for Nevow
Reading http://pypi.python.org/simple/Nevow/
Reading http://divmod.org/trac/wiki/DivmodNevow
Reading http://www.divmod.org/
Reading http://divmod.org/projects/nevow
Best match: Nevow 0.9.33
Downloading http://divmod.org/trac/attachment/wiki/SoftwareReleases/Nevow-0.9.33.tar.gz?format=raw
Processing Nevow-0.9.33.tar.gz
Running Nevow-0.9.33/setup.py -q bdist_egg --dist-dir /tmp/easy_install-kCBNGE/Nevow-0.9.33/egg-dist-tmp-ZpDGXb
Traceback (most recent call last):
  File "setup.py", line 2, in <module>
    setuptools.setup(name="minproj_that_needs_nevow", install_requires=["Nevow"])
  File "/usr/lib/python2.6/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c7.egg/setuptools/command/install.py", line 76, in run
    self.do_egg_install()
  File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c7.egg/setuptools/command/install.py", line 100, in do_egg_install
    cmd.run()
  File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c7.egg/setuptools/command/easy_install.py", line 211, in run
    self.easy_install(spec, not self.no_deps)
  File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c7.egg/setuptools/command/easy_install.py", line 427, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c7.egg/setuptools/command/easy_install.py", line 473, in install_item
    self.process_distribution(spec, dist, deps)
  File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c7.egg/setuptools/command/easy_install.py", line 519, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c7.egg/pkg_resources.py", line 522, in resolve
    dist = best[req.key] = env.best_match(req, self, installer)
  File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c7.egg/pkg_resources.py", line 758, in best_match
    return self.obtain(req, installer) # try and download/install
  File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c7.egg/pkg_resources.py", line 770, in obtain
    return installer(requirement)
  File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c7.egg/setuptools/command/easy_install.py", line 446, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c7.egg/setuptools/command/easy_install.py", line 471, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c7.egg/setuptools/command/easy_install.py", line 655, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c7.egg/setuptools/command/easy_install.py", line 930, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c7.egg/setuptools/command/easy_install.py", line 919, in run_setup
    run_setup(setup_script, args)
  File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c7.egg/setuptools/sandbox.py", line 27, in run_setup
    lambda: execfile(
  File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c7.egg/setuptools/sandbox.py", line 63, in run
    return func()
  File "/usr/local/lib/python2.6/dist-packages/setuptools-0.6c7.egg/setuptools/sandbox.py", line 29, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 3, in <module>

  File "/tmp/easy_install-kCBNGE/Nevow-0.9.33/nevow/__init__.py", line 5, in <module>
    # package placeholder
  File "/tmp/easy_install-kCBNGE/Nevow-0.9.33/nevow/_version.py", line 2, in <module>
ImportError: No module named twisted.python

This is, I believe similar to the test for nevow install that the nevow buildbot currently does, e.g.: http://buildbot.divmod.org/builders/linux32-py2.5-nevowinstall

Uninstalling Nevow from the build machine is probably not feasible. :( Fortunately, changing the dependency declaration so that it requires a version of Nevow greater than the one installed on that machine has much the same effect, as far as I can tell. Alas, the next difficulty comes in two ways:

• I am reluctant to allow the build slave to download a version of Nevow from the internet.
• Actually testing the trunk version of Nevow requires... actually using that version in the tests; even ignoring the previous point, this would require uploading a new Nevow version to PyPI for every trunk commit.

I mistakenly believed I could exercise this code path by building an egg of nevow and then using easy_install's -f option to cause it to be used. This works, however it fails to reproduce the problem, since the problem only happens if the egg is built as part of the dependency resolution process.

The only solution I can think of would involve running a PyPI-alike service and directing the builder's easy_install invocation at it using the -i option. This sounds like a lot of work, and is ridiculously complicated to boot.

Perhaps there is some other way to trigger the egg building, based on the locally available source?

It seems that putting an sdist .tar.gz into the -f directory will trigger this behavior. I started down that road, but then discovered that r17442 broke Nevow sdist. Working on that for now.

Okay. I fixed sdist and managed to finish setting up this builder. However, it still doesn't fail. Looking at the error again, I wonder if Twisted also has to be not installed for this case to come up?

Yes, if twisted is installed then the build will succeed even though Nevow requires Twisted but doesn't programmatically declare its requirement.

That's tricky. I definitely can't uninstall Twisted.

What if you set PYTHONPATH to exclude the directory that contains Twisted?

It's installed in /usr/lib/python2.4/site-packages. So I don't know of a way to exclude it without removing it.

(In [17532]) Branching to 'nevow-declare-twisted-dep-2629'

Okay, using virtualenv I managed to get a build/test environment lacking Twisted.

I applied the above patch, but to no avail. This makes sense, since Nevow's setup.py imports Twisted long before it invokes any setuptools code which could satisfy this dependency. What am I missing?

Hm. I talked to dpeterson on #distutils and he seems to think this is actually impossible. A resolution to #2699 might turn the problem into one which has a solution by loosening the dependency from one at install-time to one merely at runtime, though.

Yes, I'm sorry that in the course of this thread I got confused and started thinking that fixing this would make Nevow installable without already having Twisted installed. That isn't the case -- that's the subject of #2699.

Suppose that #2699 were fixed, so that Nevow didn't require Twisted at build-time, but it still did at run-time. Then if someone installed Nevow with easy_install Nevow or with a project that declares a requirement of Nevow (such as the minproj's in the initial comment), Nevow would successfully install, but unless this ticket were fixed, Twisted would not be installed, so Nevow would not be importable and usable. Make sense? So Nevow fails the tests that you've written for this ticket, but once this ticket and #2699 are fixed, Nevow will pass those tests.

Looking at trunk/Nevow/formless/annotate.py?rev=15202 I realize that Nevow depends on zope.interface, not only because Nevow depends on Twisted and Twisted depends on zope.interface, but also because Nevow depends on zope.interface directly. So metadata about the Nevow package ought to specify that it requires Twisted and also that it requires zope.interface. (You can see how Twisted specifies that it requires zope.interface in its setup.py.)

By the way, reading back through this ticket it appears that we got confused right from the start by thinking that fixing this issue (that Nevow does not declare its dependency on Twisted in a machine-parseable way) would lead to an easy_install Nevow automatically installing Twisted. That's not the case. Once this ticket and #2699 are fixed, then easy_install Nevow will automatically install Twisted. As long as either this ticket or #2699 are not fixed, then it won't.

However, there are other reasons to explicitly declare this metadata besides installing with easy_install. Some tools may parse this metadata to determine dependencies for other reasons, such as stdeb which uses it to figure out which Debian packages would be required by a new Nevow Debian package that stdeb would create.

So, all we need to do is write a test of whether the metadata shows that Nevow requires Twisted:

>>> import pkg_resources
>>> pkg_resources.require('Nevow')
[Nevow 0.9.31-r15675 (/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Nevow-0.9.31_r15675-py2.5.egg)]

For comparison here is what it says about Twisted's dependencies:

>>> pkg_resources.require('Twisted')
[Twisted 8.2.0 (/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-macosx-10.3-i386.egg), zope.interface 3.1.0c1 (/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages)]

See that Twisted is marked as depending on zope.interface. So here is a test for this ticket!

self.failUnless(any([d for d in pkg_resources.require('Nevow') if d.as_requirement().project_name == 'Twisted']))
self.failUnless(any([d for d in pkg_resources.require('Nevow') if d.as_requirement().project_name == 'zope.interface']))

accepting this ticket to indicate that I intend to write up those two lines from comment:31 into an actual trial test case.