Blame doc/development.rst

Pierre-Yves Chibon 300c54
Development
Pierre-Yves Chibon 300c54
===========
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Get the sources
Pierre-Yves Chibon 300c54
---------------
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Anonymous:
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
::
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 7013cc
  git clone https://pagure.io/pagure.git
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Contributors:
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
::
Pierre-Yves Chibon 300c54
Pradeep CE (cep) 19e248
  git clone ssh://git@pagure.io/pagure.git
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Dependencies
Pierre-Yves Chibon 300c54
------------
Pierre-Yves Chibon 300c54
bill auger 74c1fa
Install the build dependencies of pagure:
bill auger 74c1fa
bill auger 74c1fa
::
bill auger 74c1fa
bill auger 74c1fa
  sudo dnf install git python-virtualenv libgit2-devel \
bill auger 74c1fa
                   libjpeg-devel gcc libffi-devel redhat-rpm-config
bill auger 74c1fa
bill auger 74c1fa
bill auger 74c1fa
The python dependencies of pagure are listed in the file ``requirements.txt``
Pierre-Yves Chibon 300c54
at the top level of the sources.
Pierre-Yves Chibon 300c54
bill auger 74c1fa
::
bill auger 74c1fa
bill auger 74c1fa
  virtualenv pagure_env
bill auger 74c1fa
  source ./pagure_env/bin/activate
bill auger 74c1fa
  pip install pygit2==<version found="" libgit2="" of="">.* # e.g. 0.23.*</version>
bill auger 74c1fa
  pip install -r requirements.txt
bill auger 74c1fa
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
.. note:: working in a `virtualenv <http: en="" latest="" www.virtualenv.org="">`_</http:>
Pierre-Yves Chibon 300c54
          is tricky due to the dependency on `pygit2 <http: www.pygit2.org="">`_</http:>
Pierre-Yves Chibon 300c54
          and thus on `libgit2 <https: libgit2.github.com="">`_</https:>
Pierre-Yves Chibon 300c54
          but the pygit2 `documentation has a solution for this
Pierre-Yves Chibon 300c54
          <http: install.html#libgit2-within-a-virtual-environment="" www.pygit2.org="">`_.</http:>
Pierre-Yves Chibon 300c54
Karsten Hopp e78dc3
How to run pagure
Karsten Hopp e78dc3
-----------------
Karsten Hopp e78dc3
Karsten Hopp e78dc3
There are several options when it comes to a development environment. Vagrant
Karsten Hopp e78dc3
will provide you with a virtual machine which you can develop on, you can use
Karsten Hopp e78dc3
a container to run pagure or you can install it directly on your host machine.
Karsten Hopp e78dc3
The README has detailed instructions for the different options.
Karsten Hopp e78dc3
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Run pagure for development
Pierre-Yves Chibon d5796d
--------------------------
Pierre-Yves Chibon 300c54
Adjust the configuration file (secret key, database URL, admin group...)
Pierre-Yves Chibon 300c54
See :doc:`configuration` for more detailed information about the
Pierre-Yves Chibon 300c54
configuration.
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Create the database scheme::
Pierre-Yves Chibon 300c54
bill auger 74c1fa
  ./createdb.py
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Create the folder that will receive the different git repositories:
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
::
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 4e55c5
    mkdir {repos,docs,forks,tickets,requests,remotes}
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Run the server:
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
::
Pierre-Yves Chibon 300c54
bill auger 74c1fa
    ./runserver.py
Pierre-Yves Chibon 7013cc
Pierre-Yves Chibon 8ca915
If you want to change some configuration key you can create a file, place
Pierre-Yves Chibon 8ca915
the configuration change in it and use it with
Pierre-Yves Chibon 8ca915
Pierre-Yves Chibon 8ca915
::
Pierre-Yves Chibon 8ca915
bill auger 74c1fa
    ./runserver.py -c <config_file></config_file>
Pierre-Yves Chibon 8ca915
Pierre-Yves Chibon 8ca915
For example, create the file ``config`` with in it:
Pierre-Yves Chibon 8ca915
Pierre-Yves Chibon 8ca915
::
Pierre-Yves Chibon 8ca915
Pierre-Yves Chibon 8ca915
    from datetime import timedelta
Pierre-Yves Chibon 8ca915
    # Makes the admin session longer
Pierre-Yves Chibon 8ca915
    ADMIN_SESSION_LIFETIME = timedelta(minutes=20000000)
Pierre-Yves Chibon 8ca915
Pierre-Yves Chibon 8ca915
    # Use a postgresql database instead of sqlite
Pierre-Yves Chibon 8ca915
    DB_URL = 'postgresql://user:pass@localhost/pagure'
Pierre-Yves Chibon 8ca915
    # Change the OpenID endpoint
Pierre-Yves Chibon 8ca915
    FAS_OPENID_ENDPOINT = 'https://id.stg.fedoraproject.org'
Pierre-Yves Chibon 8ca915
Pierre-Yves Chibon 8ca915
    APP_URL = '*'
Pierre-Yves Chibon 8ca915
    EVENTSOURCE_SOURCE = 'http://localhost:8080'
Pierre-Yves Chibon 8ca915
    EVENTSOURCE_PORT = '8080'
Pierre-Yves Chibon 8ca915
    DOC_APP_URL = '*'
Pierre-Yves Chibon 8ca915
René Genz 520020
    # Avoid sending email when developing
Pierre-Yves Chibon 8ca915
    EMAIL_SEND = False
Pierre-Yves Chibon 8ca915
Pierre-Yves Chibon 8ca915
and run the server with:
Pierre-Yves Chibon 8ca915
Pierre-Yves Chibon 8ca915
::
Pierre-Yves Chibon 8ca915
bill auger 74c1fa
    ./runserver.py -c config
Pierre-Yves Chibon 8ca915
Pierre-Yves Chibon 300c54
To get some profiling information you can also run it as:
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
::
Pierre-Yves Chibon 8ca915
Pierre-Yves Chibon 300c54
    ./runserver.py --profile
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
You should be able to access the server at http://localhost:5000
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Every time you save a file, the project will be automatically restarted
Sergio Durigan Junior 7fae95
so you can see your change immediately.
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon f87eeb
Create a pull-request for testing
Pierre-Yves Chibon f87eeb
----------------------------------
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon f87eeb
When working on pagure, it is pretty often that one wanted to work on a
Pierre-Yves Chibon f87eeb
feature or a bug related to pull-requests needs to create one.
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon f87eeb
Making a pull-request for development purposes isn't hard, if you remember
Pierre-Yves Chibon f87eeb
that since you're running a local instance, the git repos created in your
Pierre-Yves Chibon f87eeb
pagure instance are also local.
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon c6fea5
So here are in a few steps that one could perform to create a pull-request in a
Pierre-Yves Chibon f87eeb
local pagure instance.
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon f87eeb
* Create a project on your pagure instance, let's say it will be called ``test``
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon c6fea5
* Create a folder ``clones`` somewhere in your system (you probably do not
Pierre-Yves Chibon c6fea5
  want it in the ``repos`` folder created above, next to it is fine though)::
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon f87eeb
    mkdir clones
Pierre-Yves Chibon f87eeb
rahul Bajaj 79c23b
* Clone the repo of the ``test`` project into this ``clones`` folder and move into it::
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon f87eeb
    cd clones
Pierre-Yves Chibon f87eeb
    git clone ~/path/to/pagure/repos/test.git
rahul Bajaj 79c23b
    cd test
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon f87eeb
* Add and commit some files::
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon f87eeb
    echo "*~" > .gitignore
Pierre-Yves Chibon f87eeb
    git add .gitignore
Pierre-Yves Chibon f87eeb
    git commit -m "Add a .gitignore file"
Pierre-Yves Chibon f87eeb
    echo "BSD" > LICENSE
Pierre-Yves Chibon f87eeb
    git add LICENSE
Pierre-Yves Chibon f87eeb
    git commit -m "Add a LICENSE file"
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon f87eeb
* Push these changes::
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon c6fea5
    git push -u origin master
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon f87eeb
* Create a new branch and add a commit in it::
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon f87eeb
    git branch new_branch
Pierre-Yves Chibon f87eeb
    git checkout new_branch
Pierre-Yves Chibon f87eeb
    touch test
Pierre-Yves Chibon f87eeb
    git add test
Pierre-Yves Chibon f87eeb
    git commit -m "Add file: test"
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon f87eeb
* Push this new branch::
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon c6fea5
    git push -u origin new_branch
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon f87eeb
Then go back to your pagure instance running in your web-browser, check the
Paul W. Frields fa5b3b
``test`` project. You should see two branches: ``master`` and ``new_branch``.
Paul W. Frields fa5b3b
From there you should be able to open a new pull-request, either from the
Pierre-Yves Chibon f87eeb
front page or via the ``File Pull Request`` button in the ``Pull Requests``
Pierre-Yves Chibon f87eeb
page.
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon f87eeb
Pierre-Yves Chibon 300c54
Coding standards
Pierre-Yves Chibon 300c54
----------------
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
We are trying to make the code `PEP8-compliant
Karsten Hopp e78dc3
<http: dev="" pep-0008="" peps="" www.python.org="">`_.  There is a `flake8 tool</http:>
Karsten Hopp e78dc3
<http: flake8="" pypi="" pypi.python.org="">`_ that can automatically check</http:>
Pierre-Yves Chibon 300c54
your source.
Pierre-Yves Chibon 300c54
Karsten Hopp e78dc3
We run the source code through `black <https: black="" pypi="" pypi.python.org="">`_</https:>
Karsten Hopp e78dc3
as part of the tests, so you may have to do some adjustments or run it
Karsten Hopp e78dc3
yourself (which is simple: ``black /path/to/pagure``).
Pierre-Yves Chibon 300c54
Karsten Hopp bbc8a9
.. note:: flake8 and black are available in Fedora:
rahul Bajaj af6966
rahul Bajaj af6966
          ::
rahul Bajaj af6966
Karsten Hopp a48335
            dnf install python3-flake8 python3-black
rahul Bajaj af6966
rahul Bajaj af6966
          or
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
          ::
Pierre-Yves Chibon 300c54
Karsten Hopp a48335
            yum install python3-flake8 python3-black
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Send patch
Pierre-Yves Chibon 300c54
----------
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
The easiest way to work on pagure is to make your own branch in git, make
Pierre-Yves Chibon 300c54
your changes to this branch, commit whenever you want, rebase on master,
Pierre-Yves Chibon 300c54
whenever you need and when you are done, send the patch either by email,
Pierre-Yves Chibon 300c54
via the trac or a pull-request (using git or github).
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
The workflow would therefore be something like:
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
::
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
   git branch <my_shiny_feature></my_shiny_feature>
Pierre-Yves Chibon 300c54
   git checkout <my_shiny_feature></my_shiny_feature>
Pierre-Yves Chibon 300c54
   <work></work>
Pierre-Yves Chibon 300c54
   git commit file1 file2
Pierre-Yves Chibon 300c54
   <more work=""></more>
Pierre-Yves Chibon 300c54
   git commit file3 file4
Pierre-Yves Chibon 300c54
   git checkout master
Pierre-Yves Chibon 300c54
   git pull
Pierre-Yves Chibon 300c54
   git checkout <my_shiny_feature></my_shiny_feature>
Pierre-Yves Chibon 300c54
   git rebase master
Pierre-Yves Chibon 300c54
   git format-patch -2
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
This will create two patch files that you can send by email to submit in a ticket
Pierre-Yves Chibon 300c54
on pagure, by email or after forking the project on pagure by submitting a
Pierre-Yves Chibon 300c54
pull-request (in which case the last step above ``git format-patch -2`` is not
Pierre-Yves Chibon 300c54
needed.
Pierre-Yves Chibon 300c54
Karsten Hopp e78dc3
.. note:: Though not required, it’s a good idea to begin the commit message 
Karsten Hopp e78dc3
          with a single short (less than 50 character) line summarizing the 
Karsten Hopp e78dc3
          change, followed by a blank line and then a more thorough description. 
Karsten Hopp e78dc3
          The text up to the first blank line in a commit message is treated 
Karsten Hopp e78dc3
          as the commit title, and that title is used throughout Git. 
Karsten Hopp e78dc3
          For example, git-format-patch turns a commit into email, and it 
Karsten Hopp e78dc3
          uses the title on the Subject line and the rest of the commit in 
Karsten Hopp e78dc3
          the body.
Karsten Hopp e78dc3
          Pagure uses lines that contain only 'Fixes #number' as references
Karsten Hopp e78dc3
          to issues. If for example a commit message of a pagure patch has 
Karsten Hopp e78dc3
          a line 'Fixes #3547' and a pullrequest (PR) gets created in pagure, 
Karsten Hopp e78dc3
          this PR will be linked to from ``https://pagure.io/pagure/issue/3547``
Karsten Hopp e78dc3
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Unit-tests
Pierre-Yves Chibon 300c54
----------
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Pagure has a number of unit-tests.
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
We aim at having a full (100%) coverage of the whole code (including the
Pierre-Yves Chibon 300c54
Flask application) and of course a smart coverage as in we want to check
Pierre-Yves Chibon 300c54
that the functions work the way we want but also that they fail when we
Pierre-Yves Chibon 300c54
expect it and the way we expect it.
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Tests checking that function are failing when/how we want are as important
Pierre-Yves Chibon 300c54
as tests checking they work the way they are intended to.
Pierre-Yves Chibon 300c54
rahul Bajaj c9695a
rahul Bajaj c9695a
So here are a few steps that one could perform to run unit-tests in a
Karsten Hopp e78dc3
local pagure instance. 
rahul Bajaj c9695a
rahul Bajaj c9695a
* Install the dependencies::
rahul Bajaj c9695a
rahul Bajaj c9695a
     pip install -r tests_requirements.txt
rahul Bajaj c9695a
rahul Bajaj c9695a
* Run it::
rahul Bajaj c9695a
rahul Bajaj c9695a
     ./runtests.sh
rahul Bajaj c9695a
rahul Bajaj c9695a
Pierre-Yves Chibon 300c54
``runtests.sh``, located at the top of the sources, helps to run the
Pierre-Yves Chibon 300c54
unit-tests of the project with coverage information using `python-nose
Pierre-Yves Chibon 300c54
<https: nose.readthedocs.org="">`_.</https:>
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
.. note:: You can specify additional arguments to the nose command used
Pierre-Yves Chibon 300c54
          in this script by just passing arguments to the script.
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
          For example you can specify the ``-x`` / ``--stop`` argument:
Pierre-Yves Chibon 300c54
          `Stop running tests after the first error or failure` by just doing
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
          ::
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
            ./runtests.sh --stop
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
Each unit-tests files (located under ``tests/``) can be called
Pierre-Yves Chibon 300c54
by alone, allowing easier debugging of the tests. For example:
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
::
Pierre-Yves Chibon 300c54
rahul Bajaj 889973
  python tests/test_pagure_lib.py
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 52c076
or using:
Pierre-Yves Chibon 52c076
Pierre-Yves Chibon 52c076
::
Pierre-Yves Chibon 52c076
Pierre-Yves Chibon 52c076
  ./runtests.sh tests/test_pagure_lib.py
Pierre-Yves Chibon 52c076
Pierre-Yves Chibon 52c076
If you with to run a single test, you can use the following structure:
Pierre-Yves Chibon 52c076
Pierre-Yves Chibon 52c076
::
Pierre-Yves Chibon 52c076
Pierre-Yves Chibon 52c076
  ./runtests.sh <file>:ClassName.method</file>
Pierre-Yves Chibon 52c076
Pierre-Yves Chibon 52c076
For example:
Pierre-Yves Chibon 52c076
Pierre-Yves Chibon 52c076
::
Pierre-Yves Chibon 52c076
Pierre-Yves Chibon 52c076
  ./runtests.sh tests/test_pagure_lib.py:PagureLibtests_search_user.test_search_user_username
Pierre-Yves Chibon 52c076
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
.. note:: In order to have coverage information you might have to install
Pierre-Yves Chibon 300c54
          ``python-coverage``
Pierre-Yves Chibon 300c54
Pierre-Yves Chibon 300c54
          ::
Pierre-Yves Chibon 300c54
rahul Bajaj af6966
            dnf install python-coverage
rahul Bajaj af6966
rahul Bajaj af6966
          or
rahul Bajaj af6966
rahul Bajaj af6966
          ::
rahul Bajaj af6966
Pierre-Yves Chibon 300c54
            yum install python-coverage
Karsten Hopp e78dc3
Karsten Hopp e78dc3
To run the unit-tests, there is also a container available with all the dependencies needed.
Karsten Hopp e78dc3
Use the following command to run the tests ::
Karsten Hopp e78dc3
Karsten Hopp e78dc3
    $ ./dev/run-tests-docker.py
Karsten Hopp e78dc3
Karsten Hopp e78dc3
This command will build a fedora based container and execute the test suite. You can also
Karsten Hopp e78dc3
limit the tests to unit-test files or single tests similar to the ``./runtests.sh`` 
Karsten Hopp e78dc3
options described above.
Karsten Hopp e78dc3
Karsten Hopp e78dc3