Initial commit of the bbuilder code and README
This commit is contained in:
parent
0020f31441
commit
9a1f5c6b6d
108
README.md
108
README.md
|
|
@ -0,0 +1,108 @@
|
|||
# Basic Builder
|
||||
|
||||
Basic Builder (bbuilder) is an extraordinarily simple CI tool. Send it webhooks from a Git management system (e.g. Gitea) and it will perform a basic set of tasks on your repository in response.
|
||||
|
||||
Tasks are specified in the `.bbuilder-tasks.yaml` file in the root of the repository. The specific events to operate on can be configured both from the webhook side, as well as via the YAML configuration inside the repository. A task can be any shell command which can be run by Python's `os.system` command.
|
||||
|
||||
## How Basic Builder works
|
||||
|
||||
At its core Basic Builder has two parts: a Flask API for handling webhook events, and a Celery worker daemon for executing the tasks.
|
||||
|
||||
The Flask API portion listens for webhooks from a compatible Git system, and the Celery worker then takes that request, clones the repository to a working directory, checks out the relevant `ref`, reads the tasks from `.bbuilder-tasks.yaml` for the current event type, and then executes them sequentially.
|
||||
|
||||
## Dependencies
|
||||
|
||||
1. Redis or another Celery-compatible broker system is required for the communication between the API and the worker(s).
|
||||
|
||||
1. `click`
|
||||
|
||||
1. `pyyaml`
|
||||
|
||||
1. `flask`
|
||||
|
||||
1. `celery`
|
||||
|
||||
The Python dependencies can be installed via `pip install -r requirements.txt`.
|
||||
|
||||
## Using Basic Builder
|
||||
|
||||
1. Help, as with all Click applications, can be shown with the `-h`/`--help` flags:
|
||||
|
||||
```
|
||||
$ bbuilder --help
|
||||
Usage: bbuilder [OPTIONS] COMMAND [ARGS]...
|
||||
|
||||
Basic Builder (bbuilder) CLI
|
||||
|
||||
Options:
|
||||
--version
|
||||
-b, --broker TEXT Celery broker URI. Envvar: BB_BROKER [default: redis://127.0.0.1:6379/0]
|
||||
-w, --work-dir TEXT Directory to perform build tasks. Envvar: BB_WORK_DIR [default: /tmp/bbuilder]
|
||||
-d, --debug Enable debug mode. Envvar: BB_DEBUG
|
||||
-h, --help Show this message and exit.
|
||||
|
||||
Commands:
|
||||
run Run the Basic Builder server.
|
||||
worker Run a Basic Builder worker.
|
||||
```
|
||||
|
||||
1. Run the API with the following command:
|
||||
|
||||
```
|
||||
$ bbuilder run
|
||||
```
|
||||
|
||||
By default, the API will listen on `0.0.0.0:7999`; you may change this with the `-a`/`--listen-addr` and `-p`/`--listen-port` options, for example:
|
||||
|
||||
```
|
||||
$ bbuilder --listen-addr 127.0.0.1 --listen-port 4000
|
||||
```
|
||||
|
||||
1. Run a worker with the following command:
|
||||
|
||||
```
|
||||
$ bbuilder.py worker
|
||||
```
|
||||
|
||||
**NOTE:** The worker runs with `concurrency=1` by default, so all tasks will be run sequentially in the order they are sent. To allow for higher load, consider setting the `-c`/`--concurrency` setting to a higher value. Note however that this may cause some tasks, for instance during release creation, to occur out of order.
|
||||
|
||||
1. Configure your Git system to send webhooks for the event(s) and repositories you want to the Basic Builder API.
|
||||
|
||||
## Webhook Configuration
|
||||
|
||||
**NOTE:** Currently, Basic Builder supports only Gitea webhooks. However, other systems may be supported in the future.
|
||||
|
||||
Webhooks are sent to Basic Builder in JSON POST format, i.e. `POST` method and `application/json` content type.
|
||||
|
||||
Normally, Basic Builder should be sent only "Repository Events" type webhook events, and only the following events are handled by Basic Builder:
|
||||
|
||||
* "Push": A normal push to a branch.
|
||||
|
||||
* "Create": The creation of a tag.
|
||||
|
||||
* "Release": The creation or editing of a release.
|
||||
|
||||
**NOTE:** Basic Builder is, as the name implies, extremely basic. These 3 are very likely the only 3 event types we will support. If you require more, a more complex CI system is what you're looking for.
|
||||
|
||||
## `.bbuilder-tasks.yaml`
|
||||
|
||||
Within each repository configured for Basic Builder must be a `.bbuilder-tasks.yaml` configuration.
|
||||
|
||||
For example, the following `.bbuilder-tasks.yaml` specifies a simple set of `echo` tasks to run on `push`, Tag `create`, and Release `published` events:
|
||||
|
||||
```
|
||||
bbuilder:
|
||||
push:
|
||||
- echo pushed
|
||||
create:
|
||||
- echo created
|
||||
release:
|
||||
published:
|
||||
- echo published
|
||||
```
|
||||
|
||||
You can extrapolate from here how to leverage Basic Builder to perform other tasks you may want on your repository. Each section is optional; if Basic Builder doesn't find any relevant tasks, it simply won't execute anything.
|
||||
|
||||
**NOTE:** The commands specified in `.bbuilder-tasks.yaml` are always run relative to the root of the repository on the relevant `ref`, either a branch for `push` events, or a tag for `create` or `release` events.
|
||||
|
||||
**NOTE:** The commands specified in `.bbuilder-tasks.yaml` are run with the privileges of the `bbuilder worker` process. Normally, this should not be `root`, but if it does need to be, **be very careful and remember that Basic Builder is implicitly trusting the content of this configuration in all repositories it is configured for**.
|
||||
|
|
@ -0,0 +1,130 @@
|
|||
#!/usr/bin/env python3
|
||||
|
||||
import click
|
||||
import bbuilder.lib.worker
|
||||
|
||||
from flask import Flask, request
|
||||
from celery import Celery
|
||||
|
||||
DEFAULT_REDIS_QUEUE = 'redis://127.0.0.1:6379/0'
|
||||
DEFAULT_WORK_DIR = '/tmp/bbuilder'
|
||||
|
||||
CONTEXT_SETTINGS = dict(help_option_names=['-h', '--help'], max_content_width=120)
|
||||
|
||||
config = dict()
|
||||
|
||||
# Version function
|
||||
def print_version(ctx, param, value):
|
||||
if not value or ctx.resilient_parsing:
|
||||
return
|
||||
from pkg_resources import get_distribution
|
||||
version = get_distribution('bbuilder').version
|
||||
click.echo(f'Basic Builder version {version}')
|
||||
ctx.exit()
|
||||
|
||||
|
||||
# Worker CLI entrypoint
|
||||
@click.command(name='worker', short_help='Run a Basic Builder worker.')
|
||||
@click.option(
|
||||
'-c', '--concurrency', 'concurrency', envvar='BB_CONCURRENCY',
|
||||
default=1, show_default=True,
|
||||
help='The concurrency of the Celery worker. Envvar: BB_CONCURRENCY'
|
||||
)
|
||||
def cli_worker(concurrency):
|
||||
"""
|
||||
Run a Basic Builder worker
|
||||
"""
|
||||
celery = Celery('bbuilder', broker=config['broker'])
|
||||
|
||||
@celery.task(bind=True)
|
||||
def do_task(self, hooktype, flask_request):
|
||||
return bbuilder.lib.worker.do_task(self, config, hooktype, flask_request)
|
||||
|
||||
worker = celery.Worker(debug=config['debug'], concurrency=concurrency)
|
||||
worker.start()
|
||||
|
||||
|
||||
# Run CLI entrypoint
|
||||
@click.command(name='run', short_help='Run the Basic Builder server.')
|
||||
@click.option(
|
||||
'-a', '--listen-addr', 'listen_addr', envvar='BB_LISTEN_ADDR',
|
||||
default='0.0.0.0', show_default=True,
|
||||
help='Listen on this address. Envvar: BB_LISTEN_ADDR'
|
||||
)
|
||||
@click.option(
|
||||
'-p', '--listen-port', 'listen_port', envvar='BB_LISTEN_PORT',
|
||||
default='7999', show_default=True,
|
||||
help='Listen on this port. Envvar: BB_LISTEN_PORT'
|
||||
)
|
||||
def cli_run(listen_addr, listen_port):
|
||||
"""
|
||||
Run the Basic Builder server
|
||||
"""
|
||||
app = Flask(__name__)
|
||||
app.config['CELERY_BROKER_URL'] = config['broker']
|
||||
app.config['CELERY_BROKER_BACKEND'] = config['broker']
|
||||
|
||||
celery = Celery('bbuilder', broker=config['broker'])
|
||||
|
||||
@app.route('/event/<hooktype>', methods=['POST'])
|
||||
def gitea_event(hooktype):
|
||||
request_json = request.get_json()
|
||||
request_headers = dict(request.headers)
|
||||
if request_json is None or request_headers is None:
|
||||
return False
|
||||
flask_request = (request_headers, request_json)
|
||||
|
||||
@celery.task(bind=True)
|
||||
def do_task(self, hooktype, flask_request):
|
||||
return bbuilder.lib.worker.do_task(self, config, hooktype, flask_request)
|
||||
|
||||
task = do_task.delay(hooktype, flask_request)
|
||||
|
||||
return { "task_id": task.id }, 202
|
||||
|
||||
app.run(listen_addr, listen_port, threaded=True, debug=config['debug'])
|
||||
|
||||
|
||||
# Main CLI entrypoint
|
||||
@click.group(context_settings=CONTEXT_SETTINGS)
|
||||
@click.option(
|
||||
'--version', is_flag=True, callback=print_version,
|
||||
expose_value=False, is_eager=True
|
||||
)
|
||||
@click.option(
|
||||
'-b', '--broker', 'broker', envvar='BB_BROKER',
|
||||
default=DEFAULT_REDIS_QUEUE, show_default=True,
|
||||
help='Celery broker URI. Envvar: BB_BROKER'
|
||||
)
|
||||
@click.option(
|
||||
'-w', '--work-dir', 'workdir', envvar='BB_WORK_DIR',
|
||||
default=DEFAULT_WORK_DIR, show_default=True,
|
||||
help='Directory to perform build tasks. Envvar: BB_WORK_DIR'
|
||||
)
|
||||
@click.option(
|
||||
'-d', '--debug', 'debug', envvar='BB_DEBUG',
|
||||
default=False, is_flag=True,
|
||||
help='Enable debug mode. Envvar: BB_DEBUG'
|
||||
)
|
||||
def cli(broker, workdir, debug):
|
||||
"""
|
||||
Basic Builder (bbuilder) CLI
|
||||
"""
|
||||
global config
|
||||
config['broker'] = broker
|
||||
config['workdir'] = workdir
|
||||
config['debug'] = debug
|
||||
|
||||
|
||||
cli.add_command(cli_worker)
|
||||
cli.add_command(cli_run)
|
||||
|
||||
#
|
||||
# Main entrypoint
|
||||
#
|
||||
def main():
|
||||
return cli(obj={})
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
|
@ -0,0 +1,161 @@
|
|||
#!/usr/bin/env python3
|
||||
|
||||
import contextlib
|
||||
import os
|
||||
import json
|
||||
import yaml
|
||||
from shutil import rmtree
|
||||
from celery import states
|
||||
from celery.exceptions import Ignore
|
||||
|
||||
|
||||
class TaskFailure(Exception):
|
||||
pass
|
||||
|
||||
|
||||
#
|
||||
# Workdir context manager
|
||||
#
|
||||
@contextlib.contextmanager
|
||||
def create_workdir(config, task_id):
|
||||
workdir_base = config['workdir']
|
||||
workdir = f"{workdir_base}/{task_id}"
|
||||
|
||||
cwd = os.getcwd()
|
||||
|
||||
os.makedirs(workdir, exist_ok=True)
|
||||
os.chdir(workdir)
|
||||
|
||||
yield workdir
|
||||
|
||||
os.chdir(cwd)
|
||||
rmtree(workdir)
|
||||
|
||||
|
||||
def handle_event_gitea(request):
|
||||
event = request[0].get('X-Gitea-Event', None)
|
||||
if event is None:
|
||||
meta = f'FATAL: No X-Gitea-Event header in request'
|
||||
raise TaskFailure(meta)
|
||||
|
||||
request_json = request[1]
|
||||
request_data = json.dumps(request_json, indent=2)
|
||||
print(f'Request JSON:\n{request_data}')
|
||||
|
||||
# Get the repository clone_url
|
||||
repository = request_json.get('repository', None)
|
||||
if repository is None:
|
||||
meta = f'FATAL: No repository information in request JSON body'
|
||||
raise TaskFailure(meta)
|
||||
clone_url = repository.get('clone_url')
|
||||
|
||||
# Get the event action (only relevant to Release events)
|
||||
event_action = request_json.get('action', None)
|
||||
|
||||
# A push event which has a specific branch as its ref
|
||||
if event == 'push':
|
||||
ref = request_json.get('ref', None)
|
||||
if ref is None:
|
||||
meta = f'FATAL: No "ref" in request JSON body'
|
||||
raise TaskFailure(meta)
|
||||
|
||||
# A release create event has a ref_type, usually "tag", and a ref (tag name)
|
||||
elif event == 'create':
|
||||
ref_type = request_json.get('ref_type', None)
|
||||
ref_name = request_json.get('ref', None)
|
||||
if ref_type is None or ref_name is None:
|
||||
meta = f'FATAL: No "ref" or "ref_type" in request JSON body'
|
||||
raise TaskFailure(meta)
|
||||
|
||||
if ref_type == 'tag':
|
||||
ref_type = 'tags'
|
||||
ref = f'refs/{ref_type}/{ref_name}'
|
||||
|
||||
# A release publish event has a release section with a tag_name
|
||||
elif event == 'release':
|
||||
release_detail = request_json.get('release', None)
|
||||
if release_detail is None:
|
||||
meta = f'FATAL: No "release" in request JSON body'
|
||||
raise TaskFailure(meta)
|
||||
|
||||
tag_name = release_detail.get('tag_name', None)
|
||||
if tag_name is None:
|
||||
meta = f'FATAL: No "tag_name" in release information'
|
||||
raise TaskFailure(meta)
|
||||
|
||||
ref = f'refs/tags/{tag_name}'
|
||||
|
||||
if event_action is None:
|
||||
meta = f'FATAL: Event is "release" but no "action" present in JSON body'
|
||||
raise TaskFailure(meta)
|
||||
|
||||
return event, event_action, clone_url, ref
|
||||
|
||||
|
||||
def clone_repository(clone_url):
|
||||
print(f"Cloning repository...")
|
||||
os.system(f'git clone {clone_url} repo')
|
||||
|
||||
|
||||
def parse_config(event, event_action):
|
||||
print(f'Parsing config from ".bbuilder-tasks.yaml"...')
|
||||
with open('.bbuilder-tasks.yaml', 'r') as fh:
|
||||
bbuilder_config = yaml.load(fh, Loader=yaml.BaseLoader).get('bbuilder', None)
|
||||
|
||||
if bbuilder_config is None:
|
||||
meta = f'FATAL: Repository ".bbuilder-tasks.yaml" does not contain valid bbuilder syntax'
|
||||
raise TaskFailure(meta)
|
||||
|
||||
tasks = bbuilder_config.get(event, [])
|
||||
# For release events, we require sub-categories for the actual event type
|
||||
if event == 'release':
|
||||
tasks = tasks.get(event_action, [])
|
||||
print(f'Tasks to perform for event "{event}.{event_action}":')
|
||||
else:
|
||||
print(f'Tasks to perform for event "{event}":')
|
||||
|
||||
for task in tasks:
|
||||
print(f"- {task}")
|
||||
|
||||
return tasks
|
||||
|
||||
|
||||
#
|
||||
# Entrypoint
|
||||
#
|
||||
hooktype_dict = {
|
||||
'gitea': handle_event_gitea
|
||||
}
|
||||
|
||||
|
||||
def do_task(self, config, hooktype, request):
|
||||
task_id = self.request.id
|
||||
print(f"Starting task {task_id}")
|
||||
|
||||
if hooktype not in hooktype_dict.keys():
|
||||
meta = f'FATAL: Hook type "{hooktype}" is not valid.'
|
||||
raise TaskFailure(meta)
|
||||
|
||||
event, event_action, clone_url, ref = hooktype_dict.get(hooktype)(request)
|
||||
|
||||
print(f"Event type: {event}")
|
||||
print(f"Clone URL: {clone_url}")
|
||||
|
||||
with create_workdir(config, task_id) as workdir:
|
||||
print(f"Operating under {workdir}")
|
||||
|
||||
clone_repository(clone_url)
|
||||
|
||||
os.chdir('repo')
|
||||
|
||||
print(f"Check out {ref}")
|
||||
os.system(f'git checkout {ref}')
|
||||
|
||||
tasks = parse_config(event, event_action)
|
||||
|
||||
for task in tasks:
|
||||
os.system(task)
|
||||
|
||||
os.chdir('..')
|
||||
|
||||
print("All tasks completed")
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
pyyaml
|
||||
flask
|
||||
celery
|
||||
click
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
from setuptools import setup
|
||||
|
||||
setup(
|
||||
name='bbuilder',
|
||||
version='0.0.1',
|
||||
packages=['bbuilder', 'bbuilder.lib'],
|
||||
install_requires=[
|
||||
'Click',
|
||||
'PyYAML',
|
||||
'lxml',
|
||||
'colorama',
|
||||
'requests',
|
||||
'requests-toolbelt',
|
||||
'flask'
|
||||
],
|
||||
entry_points={
|
||||
'console_scripts': [
|
||||
'bbuilder = bbuilder.bbuilder:cli',
|
||||
],
|
||||
},
|
||||
)
|
||||
Loading…
Reference in New Issue