Blame doc/overview.rst

Pierre-Yves Chibon e68ff3
Overview
Pierre-Yves Chibon e68ff3
========
Pierre-Yves Chibon e68ff3
Pierre-Yves Chibon 18e5d3
Pagure is split over multiple components, each having their purpose and all
Pierre-Yves Chibon 45a893
but two (the core web application and its workers) being optional.
Pierre-Yves Chibon e68ff3
Pierre-Yves Chibon e68ff3
These components are:
Pierre-Yves Chibon e68ff3
Pierre-Yves Chibon e68ff3
.. contents::
Pierre-Yves Chibon e68ff3
Pierre-Yves Chibon e68ff3
Pierre-Yves Chibon dd9db6
Before going into the overall picture, one should realize that most of the
Pierre-Yves Chibon dd9db6
components listed above are optional.
Pierre-Yves Chibon dd9db6
René Genz 520020
Here is a diagram representing pagure without all the optional components:
Pierre-Yves Chibon dd9db6
Pierre-Yves Chibon dd9db6
.. image:: _static/overview_simple.png
Pierre-Yves Chibon dd9db6
        :target: _static/overview_simple.png
Pierre-Yves Chibon dd9db6
Pierre-Yves Chibon dd9db6
Pierre-Yves Chibon dd9db6
And here is a diagram of all the components together:
Pierre-Yves Chibon 2b448e
Pierre-Yves Chibon 2b448e
.. image:: _static/overview.png
Pierre-Yves Chibon 2b448e
        :target: _static/overview.png
Pierre-Yves Chibon 2b448e
Pierre-Yves Chibon e68ff3
Pagure core application
Pierre-Yves Chibon e68ff3
-----------------------
Pierre-Yves Chibon e68ff3
Pierre-Yves Chibon e68ff3
The core application is the flask application interacting with gitolite to
Pierre-Yves Chibon 18e5d3
provide a web UI to the git repositories as well as tickets and pull-requests.
Pierre-Yves Chibon e68ff3
This is the main application for the forge.
Pierre-Yves Chibon e68ff3
Pierre-Yves Chibon e68ff3
Pierre-Yves Chibon 45a893
Pagure workers
Pierre-Yves Chibon 45a893
--------------
Pierre-Yves Chibon 45a893
Pierre-Yves Chibon 45a893
Interacting with git repos can be a long process, it varies depending on the
Pierre-Yves Chibon 45a893
size of the repository itself but also based on hardware performances or
Pierre-Yves Chibon 45a893
simply the load on the system.
Pierre-Yves Chibon 45a893
To make pagure capable of handling more load, since pagure 3.0 the interactions
Pierre-Yves Chibon 2ce2bb
with the git repositories from the web UI is performed by dedicated workers,
Pierre-Yves Chibon 45a893
allowing async processing of the different tasks.
Pierre-Yves Chibon 45a893
Pierre-Yves Chibon 45a893
The communication between the core application and its worker is based on
Pierre-Yves Chibon 2ce2bb
`celery <http: www.celeryproject.org="">`_ and defaults to using `redis</http:>
Pierre-Yves Chibon 2ce2bb
<https: redis.org="">`_ but any of the queueing system supported by `celery</https:>
Pierre-Yves Chibon 2ce2bb
<http: www.celeryproject.org="">`_ could be used instead.</http:>
Pierre-Yves Chibon 45a893
Pierre-Yves Chibon 45a893
Pierre-Yves Chibon 8bbf26
Gitolite
Pierre-Yves Chibon 8bbf26
--------
Pierre-Yves Chibon 8bbf26
Pierre-Yves Chibon 8bbf26
Currently pagure uses `gitolite <http: gitolite="" gitolite.com="" index.html="">`_</http:>
Pierre-Yves Chibon 8bbf26
to grant or deny `ssh <https: en.wikipedia.org="" secure_shell="" wiki="">`_ access</https:>
Pierre-Yves Chibon 8bbf26
to the git repositories, in other words to grant or deny read and/or write
Pierre-Yves Chibon 8bbf26
access to the git repositories.
Pierre-Yves Chibon 8bbf26
Pierre-Yves Chibon 8bbf26
Pagure supports cloning over both ssh and http, but writing can only be done
Pierre-Yves Chibon 8bbf26
via ssh, through gitolite.
Pierre-Yves Chibon 8bbf26
Pierre-Yves Chibon 8bbf26
Pierre-Yves Chibon e68ff3
Pagure doc server
Pierre-Yves Chibon e68ff3
-----------------
Pierre-Yves Chibon e68ff3
Pierre-Yves Chibon e68ff3
While integrated into the main application at first, it has been split out
Pierre-Yves Chibon e68ff3
for security concern, displaying information directly provided by the user
René Genz 520020
without a clear/safe way of filtering for unsafe script or hacks is a
Pierre-Yves Chibon e68ff3
security hole.
Pierre-Yves Chibon e68ff3
For this reason we also strongly encourage anyone wanting to deploy their
Pierre-Yves Chibon 18e5d3
own instance of pagure with the doc server, to run this application on a
Pierre-Yves Chibon e68ff3
completely different domain name (not just a sub-domain) in order to reduce
Pierre-Yves Chibon e68ff3
the cross-site forgery risks.
Pierre-Yves Chibon e68ff3
Pierre-Yves Chibon e68ff3
Pagure can be run just fine without the doc server, all you need to do is to
Pierre-Yves Chibon e68ff3
**not** define the variable ``DOC_APP_URL`` in the configuration file.
Pierre-Yves Chibon e68ff3
Pierre-Yves Chibon e68ff3
Pierre-Yves Chibon e68ff3
Pagure milter
Pierre-Yves Chibon e68ff3
-------------
Pierre-Yves Chibon e68ff3
Pierre-Yves Chibon e68ff3
The milter is a script, receiving an email as input and performing an action
Pierre-Yves Chibon e68ff3
with it.
Pierre-Yves Chibon e68ff3
Pierre-Yves Chibon e68ff3
In the case of pagure, the milter is used to allow replying on a comment
Pierre-Yves Chibon e68ff3
of a ticket or a pull-request by directly replying to the notification sent.
Pierre-Yves Chibon e68ff3
No need to go to the page anymore to reply to a comment someone made.
Pierre-Yves Chibon e68ff3
Pierre-Yves Chibon fab115
The milter integrates with a MTA such as postfix or sendmail that you will
Pierre-Yves Chibon fab115
have running and have access to in order to change its configuration.
Pierre-Yves Chibon e68ff3
Pierre-Yves Chibon e68ff3
Pierre-Yves Chibon e68ff3
Pagure EventSource Server
Pierre-Yves Chibon e68ff3
-------------------------
Pierre-Yves Chibon e68ff3
Pierre-Yves Chibon e68ff3
Eventsource or Server Sent Events are messages sent from a server to a browser.
Pierre-Yves Chibon e68ff3
Pierre-Yves Chibon e68ff3
For pagure this technology is used to allow live-refreshing of a page when
Pierre-Yves Chibon e68ff3
someone is viewing it. For example, while you are reading a ticket if someone
Pierre-Yves Chibon e68ff3
comments on it, the comment will automatically show up on the page without
Pierre-Yves Chibon e68ff3
the need for you to reload the entire page.
Pierre-Yves Chibon cd8e87
Pierre-Yves Chibon d89186
The flow is: the main pagure server does an action, sends a message over
Pierre-Yves Chibon d89186
redis, the eventsource server picks it up and send it to the browsers waiting
Pierre-Yves Chibon d89186
for it, then javascript code is executed to refresh the page based on the
Pierre-Yves Chibon d89186
information received.
Pierre-Yves Chibon d89186
Pierre-Yves Chibon cd8e87
Pierre-Yves Chibon cd8e87
Pagure web-hook Server
Pierre-Yves Chibon 45a893
----------------------
Pierre-Yves Chibon cd8e87
Pierre-Yves Chibon cd8e87
Sends notifications to third party services using POST http requests.
Pierre-Yves Chibon cd8e87
Pierre-Yves Chibon cd8e87
This is the second notifications system in pagure with `fedmsg <http: fedmsg.com="">`_.</http:>
Pierre-Yves Chibon cd8e87
These notifications are running on their own service to prevent blocking the
Pierre-Yves Chibon cd8e87
main web application in case the third part service is timing-out or just
Pierre-Yves Chibon cd8e87
being slow.
Pierre-Yves Chibon d89186
Pierre-Yves Chibon d89186
The flow is: the main pagure server does an action, sends a message over
Pierre-Yves Chibon d89186
redis, the web-hook server picks it up, build the query and performs the
Pierre-Yves Chibon d89186
POST request to the specified URLs.
Pierre-Yves Chibon 45a893
Pierre-Yves Chibon 45a893
Pierre-Yves Chibon 45a893
Pagure load JSON service
Pierre-Yves Chibon 45a893
------------------------
Pierre-Yves Chibon 45a893
Pierre-Yves Chibon 45a893
The load JSON service is an async service updating the database based on
Pierre-Yves Chibon 45a893
information pushed to the ticket or pull-request git repositories.
Pierre-Yves Chibon 45a893
This allows updating the database with information pushed to the git
Pierre-Yves Chibon 45a893
repositories without keeping the connection open with the client.
Pierre-Yves Chibon 45a893
Pierre-Yves Chibon 45a893
Pierre-Yves Chibon 45a893
Pagure log com service
Pierre-Yves Chibon 45a893
----------------------
Pierre-Yves Chibon 45a893
Pierre-Yves Chibon 45a893
The log com (for log commit) service is an async service updating the log
Pierre-Yves Chibon 45a893
table of the database on every pushed made to any repository allowing to
Pierre-Yves Chibon 45a893
build the data for the calendar heatmap graph displayed on every user's
Pierre-Yves Chibon 45a893
page.