From 45a8938cdf6ee6df3de484305ca40cc92d191313 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Jun 14 2017 07:26:18 +0000 Subject: Update the documentation for the coming 3.0 release --- diff --git a/doc/configuration.rst b/doc/configuration.rst index 7e2b6c9..a2acff8 100644 --- a/doc/configuration.rst +++ b/doc/configuration.rst @@ -742,7 +742,7 @@ Defaults to: ``None`` GITOLITE_POST_CONFIG -~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~ This configuration key allows you to include some content at the *end* of the gitolite configuration file (such as some project definition or access), diff --git a/doc/index.rst b/doc/index.rst index bf330f1..d283886 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -32,11 +32,12 @@ Contents: overview usage/index install - install_milter - install_evs - install_webhooks + install_pagure_milter + install_pagure_ev + install_pagure_webhooks install_pagure_ci install_pagure_loadjson + install_pagure_logcom configuration development contributing diff --git a/doc/install_evs.rst b/doc/install_evs.rst deleted file mode 100644 index 13c95fb..0000000 --- a/doc/install_evs.rst +++ /dev/null @@ -1,48 +0,0 @@ -Installing pagure's EventSource server -====================================== - -Eventsource or Server Sent Events are messages sent from a server to a web -browser. It allows one to refresh a page "live", ie, without the need to reload -it entirely. - - -Configure your system ---------------------- - -The eventsource server is easy to set-up. - -* Install the required dependencies - -:: - - python-redis - python-trollius - python-trollius-redis - -.. note:: We ship a systemd unit file for pagure_milter but we welcome patches - for scripts for other init systems. - - -* Install the files of the SSE server as follow: - -+----------------------------------------+-----------------------------------------------------+ -| Source | Destination | -+========================================+=====================================================+ -| ``pagure-ev/pagure_stream_server.py`` | ``/usr/libexec/pagure-ev/pagure_stream_server.py`` | -+----------------------------------------+-----------------------------------------------------+ -| ``pagure-ev/pagure_ev.service`` | ``/etc/systemd/system/pagure_ev.service`` | -+----------------------------------------+-----------------------------------------------------+ - -The first file is the script of the SSE server itself. - -The second file is the systemd service file. - - -* Finally, activate the service and ensure it's started upon boot: - -:: - - systemctl enable redis - systemctl start redis - systemctl enable pagure_ev - systemctl start pagure_ev diff --git a/doc/install_milter.rst b/doc/install_milter.rst deleted file mode 100644 index f1025c9..0000000 --- a/doc/install_milter.rst +++ /dev/null @@ -1,79 +0,0 @@ -Installing pagure's milter -========================== - -A milter is a script that is ran by a Mail Transfer Agent (`MTA -`_) -upon receiving an email via either a network or an unix socket. - -If you want more information feel free to check out the corresponding page -on wikipedia: `https://en.wikipedia.org/wiki/Milter -`_. - -Configure your system ---------------------- - -* Install the required dependencies - -:: - - python-pymilter - -.. note:: We ship a systemd unit file for pagure_milter but we welcome patches - for scripts for other init systems. - -.. note:: It also requires a MTA, we used postfix. - - -* Create an alias ``reply`` - -This can be done in ``/etc/aliases``, for example: -:: - - reply: /dev/null - - -* Activate the ability of your MTA, to split users based on the character ``+``. - This way all the emails sent to ``reply+...@example.com`` will be forwarded - to your alias for ``reply``. - - -In postfix this is done via: -:: - - recipient_delimiter = + - -* Hook the milter in the MTA - -In postfix this is done via: -:: - - non_smtpd_milters = unix:/var/run/pagure/paguresock - smtpd_milters = unix:/var/run/pagure/paguresock - - -* Install the files of the milter as follow: - -+---------------------------------------------+---------------------------------------------------+ -| Source | Destination | -+=============================================+===================================================+ -| ``pagure-milters/comment_email_milter.py`` | ``/usr/share/pagure/comment_email_milter.py`` | -+---------------------------------------------+---------------------------------------------------+ -| ``pagure-milters/milter_tempfile.conf`` | ``/usr/lib/tmpfiles.d/pagure-milter.conf`` | -+---------------------------------------------+---------------------------------------------------+ -| ``pagure-milters/pagure_milter.service`` | ``/etc/systemd/system/pagure_milter.service`` | -+---------------------------------------------+---------------------------------------------------+ - -The first file is the script of the milter itself. - -The second file is a file specific for systemd and ensuring the temporary -folders needed by the milter are re-created if needed at each boot. - -The third file is the systemd service file. - - -* Activate the service and ensure it's started upon boot: - -:: - - systemctl enable pagure_milter - systemctl start pagure_milter diff --git a/doc/install_pagure_ev.rst b/doc/install_pagure_ev.rst new file mode 100644 index 0000000..13c95fb --- /dev/null +++ b/doc/install_pagure_ev.rst @@ -0,0 +1,48 @@ +Installing pagure's EventSource server +====================================== + +Eventsource or Server Sent Events are messages sent from a server to a web +browser. It allows one to refresh a page "live", ie, without the need to reload +it entirely. + + +Configure your system +--------------------- + +The eventsource server is easy to set-up. + +* Install the required dependencies + +:: + + python-redis + python-trollius + python-trollius-redis + +.. note:: We ship a systemd unit file for pagure_milter but we welcome patches + for scripts for other init systems. + + +* Install the files of the SSE server as follow: + ++----------------------------------------+-----------------------------------------------------+ +| Source | Destination | ++========================================+=====================================================+ +| ``pagure-ev/pagure_stream_server.py`` | ``/usr/libexec/pagure-ev/pagure_stream_server.py`` | ++----------------------------------------+-----------------------------------------------------+ +| ``pagure-ev/pagure_ev.service`` | ``/etc/systemd/system/pagure_ev.service`` | ++----------------------------------------+-----------------------------------------------------+ + +The first file is the script of the SSE server itself. + +The second file is the systemd service file. + + +* Finally, activate the service and ensure it's started upon boot: + +:: + + systemctl enable redis + systemctl start redis + systemctl enable pagure_ev + systemctl start pagure_ev diff --git a/doc/install_pagure_logcom.rst b/doc/install_pagure_logcom.rst new file mode 100644 index 0000000..4c4c1de --- /dev/null +++ b/doc/install_pagure_logcom.rst @@ -0,0 +1,47 @@ +Installing pagure-logcom +======================== + +pagure-logcom is the service that updates the log table in the database +for every commit made to the main branch of a repository allowing to build +the calendar heatmap presented on every user's page. + + +Configure your system +--------------------- + +* Install the required dependencies + +:: + + python-redis + python-trollius-redis + python-trollius + +.. note:: We ship a systemd unit file for pagure_logcom but we welcome patches + for scripts for other init systems. + + +* Install the files of pagure-loadjon as follow: + ++-----------------------------------------------+-------------------------------------------------------+ +| Source | Destination | ++===============================================+=======================================================+ +| ``pagure-logcom/pagure_logcom_server.py`` | ``/usr/libexec/pagure-logcom/pagure_logcom_server.py``| ++--------------------------------------------------+----------------------------------------------------+ +| ``pagure-logcom/pagure_logcom.service`` | ``/etc/systemd/system/pagure_logcom.service`` | ++-----------------------------------------------+-------------------------------------------------------+ + +The first file is the pagure-logcom service itself, triggered by the git +hook (shipped with pagure itself) and logging the commits into the database. + +The second file is the systemd service file. + + +* Activate the service and ensure it's started upon boot: + +:: + + systemctl enable redis + systemctl start redis + systemctl enable pagure_logcom + systemctl start pagure_logcom diff --git a/doc/install_pagure_milter.rst b/doc/install_pagure_milter.rst new file mode 100644 index 0000000..f1025c9 --- /dev/null +++ b/doc/install_pagure_milter.rst @@ -0,0 +1,79 @@ +Installing pagure's milter +========================== + +A milter is a script that is ran by a Mail Transfer Agent (`MTA +`_) +upon receiving an email via either a network or an unix socket. + +If you want more information feel free to check out the corresponding page +on wikipedia: `https://en.wikipedia.org/wiki/Milter +`_. + +Configure your system +--------------------- + +* Install the required dependencies + +:: + + python-pymilter + +.. note:: We ship a systemd unit file for pagure_milter but we welcome patches + for scripts for other init systems. + +.. note:: It also requires a MTA, we used postfix. + + +* Create an alias ``reply`` + +This can be done in ``/etc/aliases``, for example: +:: + + reply: /dev/null + + +* Activate the ability of your MTA, to split users based on the character ``+``. + This way all the emails sent to ``reply+...@example.com`` will be forwarded + to your alias for ``reply``. + + +In postfix this is done via: +:: + + recipient_delimiter = + + +* Hook the milter in the MTA + +In postfix this is done via: +:: + + non_smtpd_milters = unix:/var/run/pagure/paguresock + smtpd_milters = unix:/var/run/pagure/paguresock + + +* Install the files of the milter as follow: + ++---------------------------------------------+---------------------------------------------------+ +| Source | Destination | ++=============================================+===================================================+ +| ``pagure-milters/comment_email_milter.py`` | ``/usr/share/pagure/comment_email_milter.py`` | ++---------------------------------------------+---------------------------------------------------+ +| ``pagure-milters/milter_tempfile.conf`` | ``/usr/lib/tmpfiles.d/pagure-milter.conf`` | ++---------------------------------------------+---------------------------------------------------+ +| ``pagure-milters/pagure_milter.service`` | ``/etc/systemd/system/pagure_milter.service`` | ++---------------------------------------------+---------------------------------------------------+ + +The first file is the script of the milter itself. + +The second file is a file specific for systemd and ensuring the temporary +folders needed by the milter are re-created if needed at each boot. + +The third file is the systemd service file. + + +* Activate the service and ensure it's started upon boot: + +:: + + systemctl enable pagure_milter + systemctl start pagure_milter diff --git a/doc/install_pagure_webhooks.rst b/doc/install_pagure_webhooks.rst new file mode 100644 index 0000000..42531e9 --- /dev/null +++ b/doc/install_pagure_webhooks.rst @@ -0,0 +1,49 @@ +Installing pagure's web-hooks notification system +================================================= + +Web-hooks are a notification system upon which a system makes a http POST +request with some data upon doing an action. This allows notifying a system +that an action has occurred. + +If you want more information feel free to check out the corresponding page +on wikipedia: `https://en.wikipedia.org/wiki/Webhook +`_. + +Configure your system +--------------------- + +* Install the required dependencies + +:: + + python-redis + python-trollius + python-trollius-redis + +.. note:: We ship a systemd unit file for pagure_webhook but we welcome patches + for scripts for other init systems. + + +* Install the files of the web-hook server as follow: + ++----------------------------------------------+----------------------------------------------------------+ +| Source | Destination | ++==============================================+==========================================================+ +| ``pagure-webhook/pagure-webhook-server.py`` | ``/usr/libexec/pagure-webhook/pagure-webhook-server.py`` | ++----------------------------------------------+----------------------------------------------------------+ +| ``pagure-webhook/pagure_webhook.service`` | ``/etc/systemd/system/pagure_webhook.service`` | ++----------------------------------------------+----------------------------------------------------------+ + +The first file is the script of the web-hook server itself. + +The second file is the systemd service file. + + +* Activate the service and ensure it's started upon boot: + +:: + + systemctl enable redis + systemctl start redis + systemctl enable pagure_webhook + systemctl start pagure_webhook diff --git a/doc/install_webhooks.rst b/doc/install_webhooks.rst deleted file mode 100644 index 42531e9..0000000 --- a/doc/install_webhooks.rst +++ /dev/null @@ -1,49 +0,0 @@ -Installing pagure's web-hooks notification system -================================================= - -Web-hooks are a notification system upon which a system makes a http POST -request with some data upon doing an action. This allows notifying a system -that an action has occurred. - -If you want more information feel free to check out the corresponding page -on wikipedia: `https://en.wikipedia.org/wiki/Webhook -`_. - -Configure your system ---------------------- - -* Install the required dependencies - -:: - - python-redis - python-trollius - python-trollius-redis - -.. note:: We ship a systemd unit file for pagure_webhook but we welcome patches - for scripts for other init systems. - - -* Install the files of the web-hook server as follow: - -+----------------------------------------------+----------------------------------------------------------+ -| Source | Destination | -+==============================================+==========================================================+ -| ``pagure-webhook/pagure-webhook-server.py`` | ``/usr/libexec/pagure-webhook/pagure-webhook-server.py`` | -+----------------------------------------------+----------------------------------------------------------+ -| ``pagure-webhook/pagure_webhook.service`` | ``/etc/systemd/system/pagure_webhook.service`` | -+----------------------------------------------+----------------------------------------------------------+ - -The first file is the script of the web-hook server itself. - -The second file is the systemd service file. - - -* Activate the service and ensure it's started upon boot: - -:: - - systemctl enable redis - systemctl start redis - systemctl enable pagure_webhook - systemctl start pagure_webhook diff --git a/doc/overview.rst b/doc/overview.rst index 66d2e84..43cb87a 100644 --- a/doc/overview.rst +++ b/doc/overview.rst @@ -2,7 +2,7 @@ Overview ======== Pagure is split over multiple components, each having their purpose and all -but one (the core application) being optional. +but two (the core web application and its workers) being optional. These components are: @@ -31,6 +31,21 @@ provide a web UI to the git repositories as well as tickets and pull-requests. This is the main application for the forge. +Pagure workers +-------------- + +Interacting with git repos can be a long process, it varies depending on the +size of the repository itself but also based on hardware performances or +simply the load on the system. +To make pagure capable of handling more load, since pagure 3.0 the interactions +with the git repositories from the web UI is performed by dedicated workers, +allowing async processing of the different tasks. + +The communication between the core application and its worker is based on +`celery <>`_ and defaults to using `redis `_ but any of +the queueing system supported by `celery <>`_ could be used instead. + + Gitolite -------- @@ -90,7 +105,7 @@ information received. Pagure web-hook Server -------------------------- +---------------------- Sends notifications to third party services using POST http requests. @@ -102,3 +117,21 @@ being slow. The flow is: the main pagure server does an action, sends a message over redis, the web-hook server picks it up, build the query and performs the POST request to the specified URLs. + + +Pagure load JSON service +------------------------ + +The load JSON service is an async service updating the database based on +information pushed to the ticket or pull-request git repositories. +This allows updating the database with information pushed to the git +repositories without keeping the connection open with the client. + + +Pagure log com service +---------------------- + +The log com (for log commit) service is an async service updating the log +table of the database on every pushed made to any repository allowing to +build the data for the calendar heatmap graph displayed on every user's +page. diff --git a/doc/overview_simple.ascii b/doc/overview_simple.ascii index a1a129b..62fc7b0 100644 --- a/doc/overview_simple.ascii +++ b/doc/overview_simple.ascii @@ -5,22 +5,31 @@ User's git actions+--------------------->+ Gitolite +-------------------------->+ Git repos | | | | | +------------+ +-----------+ - ^ - | - +-----------------------------+ - | -User's mail client | - ^ +---------------+ | - | Notifications | | | - +----------------------------------+ Mail server | | - | | | - +---------------+ | - ^ | - | | - | | - +--------------+ Updates +--------------+ - | | & queries |{s} | -User's web browser+---------------------->+ Pagure +------------->| Database | - | web server | | | - | | +--------------+ + ^ ^ + | | + +-----------------------------+ | + | | +User's mail client | | + ^ +---------------+ | | + | Notifications | | | | + +----------------------------------+ Mail server | | | + | | | | + +---------------+ | | + ^ | | + | | | + | | | + +--------------+ Updates +--------------+ | + | | & queries |{s} | | +User's web browser+---------------------->+ Pagure +------------->| Database | | + | web server | | | | + | | +--------------+ | + +------+-------+ | + | | + | sends instructions | Updates + v | + +--------------+ | + | | | + | Pagure +----------------------------------+ + | workers | + | | +--------------+ diff --git a/doc/usage/ticket_templates.rst b/doc/usage/ticket_templates.rst index 625d2ff..07e6e9f 100644 --- a/doc/usage/ticket_templates.rst +++ b/doc/usage/ticket_templates.rst @@ -10,7 +10,7 @@ often requested/needed. The templates are provided in the git repository containing the meta-data for the tickets. They must be placed under a ``templates`` folder in this git repository, -end with ``.md``and as the extension suggests can be formatted as markdown. +end with ``.md`` and as the extension suggests can be formatted as markdown. If you create a template ``templates/default.md``, it will be shown by default when someone ask to create a new ticket.