Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
967d9f3
fix error url
May 21, 2018
b83be9d
[plugins] Simplify usage of plugins with "default" platform
andreykurilin May 22, 2018
386550a
Remove hardcode to openstack from required_platform validator
andreykurilin May 22, 2018
99264ea
Drop all qpid related explanations
May 29, 2018
1e0cad1
[env] Fix validation existing platforms in py3
andreykurilin May 31, 2018
d70140a
Merge "[env] Fix validation existing platforms in py3"
May 31, 2018
b41f282
Merge "Drop all qpid related explanations"
Jun 5, 2018
97f2bd8
Amend the spelling error of a word
chengebj5238 Jun 7, 2018
42317ad
Fix typos
baiwenteng Jun 11, 2018
e70ee4b
Merge "[plugins] Simplify usage of plugins with "default" platform"
Jun 13, 2018
c77ce40
Merge "Remove hardcode to openstack from required_platform validator"
Jun 13, 2018
85d7eb3
[evil][ci] Preparing to remove all OpenStack in-tree plugins
andreykurilin Jun 13, 2018
86cd7df
[verifiers] Fix identification of testr/stestr
andreykurilin Jun 20, 2018
7addc52
[evil] Remove OpenStack related plugins
andreykurilin May 8, 2018
5dfda15
Prepare to Rally 1.0.0 release
andreykurilin Jun 20, 2018
0813f35
fix typos
sunqingliang6 Jun 21, 2018
044896e
Add py36 testenv
Jun 25, 2018
e2befff
Remove redundant message about openstack plugins
andreykurilin Jul 1, 2018
1deae09
Update config sample file
andreykurilin Jul 2, 2018
dd47162
Invalid links in doc
Jul 5, 2018
c35fff1
Merge "fix typos"
Jul 5, 2018
f5e7d3d
Merge "Invalid links in doc"
Jul 5, 2018
58c95eb
Fix minor typo in Rally Plugins doc
u6768 Jul 11, 2018
8b58142
Extend json exporter and fix elastic exporter
andreykurilin Jul 12, 2018
cbd4018
Merge "Extend json exporter and fix elastic exporter"
Jul 17, 2018
2809ab2
Merge "Add py36 testenv"
Jul 20, 2018
3a765fc
[trivial] Update task exporter
andreykurilin Jul 24, 2018
0066da8
Add filter option into `rally task detailed`
chenhb-zte Nov 30, 2017
1996507
[ci] Fix floating-bug in functional tests
andreykurilin Jul 26, 2018
a972fc9
Merge "Add filter option into `rally task detailed`"
Jul 26, 2018
95d25e5
Update the new feature of `rally task detailed` into CHANGELOG
chenhb-zte Jul 26, 2018
a7449db
Merge "Update the new feature of `rally task detailed` into CHANGELOG"
Jul 31, 2018
0f8b605
Merge "[ci] Fix floating-bug in functional tests"
Jul 31, 2018
6888a15
Add rally job for noiro
tbachman May 18, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
93 changes: 93 additions & 0 deletions CHANGELOG.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
=========
Changelog
=========

.. Changelogs are for humans, not machines. The end users of Rally project are
human beings who care about what's is changing, why and how it affects them.
Please leave these notes as much as possible human oriented.

.. Each release can use the next sections:

- **Added** for new features.
- **Changed** for changes in existing functionality.
- **Deprecated** for soon-to-be removed features/plugins.
- **Removed** for now removed features/plugins.
- **Fixed** for any bug fixes.

.. Release notes for existing releases are MUTABLE! If there is something that
was missed or can be improved, feel free to change it!

[Unreleased]
------------

Changed
~~~~~~~

* The output of json task result exporter (``rally task report --json``) is
extended with information about environment where task was executed (new
``env_name`` and ``env_uuid`` properties)
* Add the --filter-by option to the command ``rally task detailed``, which
allows us to show only those workloads which we are interested in (see the
examples below).
Examples:
1. show only failed workloads
``rally task detailed --filter-by sla-failures``
2. show only those workloads which include the next scenario plugin(s)
``rally task detailed --filter-by scenarios=scenario1[,scenarios2...]``

Removed
~~~~~~~

* Disturbing warning message about removing in-tree OpenStack plugins. This
message became redundant after Rally 1.0.0 when such plugins were removed.
* OpenStack related configuration options for sample file.
* Deprecated in Rally 0.10 ``rally.task.exporter.Exporter`` class in favor of
``rally.task.exporter.TaskExporter``.

[1.0.0] - 2018-06-20
--------------------

It finally happened. We are happy to inform you that OpenStack plugins has a
single home - https://github.com/openstack/rally-openstack .
All in-tree plugins are removed now and framework part become more lightweight.

What does it mean for you?!
~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you are interested only in OpenStack plugins, just change the package you
are installing from ``rally`` to ``rally-openstack``. If you have custom
OpenStack plugins which inherits from upstream, change python imports from
``rally.plugins.openstack`` to ``rally_openstack``. That is all.

If you are interested not only in OpenStack, you can start using your favourite
tool for various platforms and systems. Here you can find our first attempts
to seize the world - https://github.com/xrally/xrally-docker and
https://github.com/xrally/xrally-kubernetes.

Changed
~~~~~~~

Since OpenStack plugins were moved to the separate repository, the new release
notes should become light as well, so there is no need in separate pages for
each release. All release notes will be aggregated in
`a single file CHANGELOG.rst
<https://github.com/openstack/rally/blob/master/CHANGELOG.rst>`_.

Also, it is sad to mention, but due to OpenStack policies we need to stop
duplicating release notes at ``git tag message``. At least for now.

Removed
~~~~~~~

* All OpenStack related plugins.

Fixed
~~~~~

* Validation of existing platforms in Python 3 environment.
* Support of testr for verifiers.

[0.0.0] - [0.12.1]
------------------

Release notes for Rally ``0.0.0``-``0.12.1`` are available at
https://github.com/openstack/rally/tree/master/doc/release_notes/archive
53 changes: 33 additions & 20 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@
Rally
=====

Rally is tool & framework that allows one to write simple plugins and combine
them in complex tests scenarios that allows to perform all kinds of testing!

Team and repository tags
========================

Expand All @@ -28,14 +31,9 @@ Team and repository tags
What is Rally
=============

Rally is a Benchmark-as-a-Service project for OpenStack.

Rally is intended to provide the community with a benchmarking tool that is capable of performing **specific**, **complicated** and **reproducible** test cases on **real deployment** scenarios.

If you are here, you are probably familiar with OpenStack and you also know that it's a really huge ecosystem of cooperative services. When something fails, performs slowly or doesn't scale, it's really hard to answer different questions on "what", "why" and "where" has happened. Another reason why you could be here is that you would like to build an OpenStack CI/CD system that will allow you to improve SLA, performance and stability of OpenStack continuously.

The OpenStack QA team mostly works on CI/CD that ensures that new patches don't break some specific single node installation of OpenStack. On the other hand it's clear that such CI/CD is only an indication and does not cover all cases (e.g. if a cloud works well on a single node installation it doesn't mean that it will continue to do so on a 1k servers installation under high load as well). Rally aims to fix this and help us to answer the question "How does OpenStack work at scale?". To make it possible, we are going to automate and unify all steps that are required for benchmarking OpenStack at scale: multi-node OS deployment, verification, benchmarking & profiling.

Rally is intended to provide a testing framework that is
capable to perform **specific**, **complicated** and **reproducible**
test cases on **real deployment** scenarios.

**Rally** workflow can be visualized by the following diagram:

Expand All @@ -53,18 +51,30 @@ Who Is Using Rally
Documentation
=============

`Rally documentation on ReadTheDocs <https://rally.readthedocs.org/en/latest/>`_ is a perfect place to start learning about Rally. It provides you with an **easy** and **illustrative** guidance through this benchmarking tool. For example, check out the `Rally step-by-step tutorial <https://rally.readthedocs.io/en/latest/quick_start/tutorial.html>`_ that explains, in a series of lessons, how to explore the power of Rally in benchmarking your OpenStack clouds.
`Rally documentation on ReadTheDocs <https://rally.readthedocs.org/en/latest/>`_
is a perfect place to start learning about Rally. It provides you with an
**easy** and **illustrative** guidance through this benchmarking tool.

For example, check out the `Rally step-by-step tutorial
<https://rally.readthedocs.io/en/latest/quick_start/tutorial.html>`_ that
explains, in a series of lessons, how to explore the power of Rally in
benchmarking your OpenStack clouds.

Architecture
------------

In terms of software architecture, Rally is built of 4 main components:

1. **Server Providers** - provide servers (virtual servers), with ssh access, in one L3 network.
2. **Deploy Engines** - deploy OpenStack cloud on servers that are presented by **Server Providers**
3. **Verification** - component that runs tempest (or another specific set of tests) against a deployed cloud, collects results & presents them in human readable form.
4. **Benchmark engine** - allows to write parameterized benchmark scenarios & run them against the cloud.
1. **Environment** - one of key component in Rally. It manages and stores
information about tested platforms. Env manager is using platform plugins
to: create, delete, cleanup, check health, obtain information about
platforms.
2. **Task** component is responsible for executing tests defined in
task specs, persisting and reporting results.
3. **Verification** component allows to wrap subunit-based testing tools and
provide complete tool on top of them with allow to do pre configuration,
post cleanup as well process and persist results to Rally DB for future use
like reporting and results comparing.

Use Cases
---------
Expand All @@ -77,26 +87,29 @@ There are 3 major high level Rally Use Cases:

Typical cases where Rally aims to help are:

- Automate measuring & profiling focused on how new code changes affect the OS performance;
- Automate measuring & profiling focused on how new code changes affect the
OpenStack performance;
- Using Rally profiler to detect scaling & performance issues;
- Investigate how different deployments affect the OS performance:
- Find the set of suitable OpenStack deployment architectures;
- Create deployment specifications for different loads (amount of controllers, swift nodes, etc.);

- Find the set of suitable OpenStack deployment architectures;
- Create deployment specifications for different loads (amount of
controllers, swift nodes, etc.);
- Automate the search for hardware best suited for particular OpenStack cloud;
- Automate the production cloud specification generation:
- Determine terminal loads for basic cloud operations: VM start & stop, Block Device create/destroy & various OpenStack API methods;
- Check performance of basic cloud operations in case of different loads.

- Determine terminal loads for basic cloud operations: VM start & stop,
Block Device create/destroy & various OpenStack API methods;
- Check performance of basic cloud operations in case of different loads.

Links
----------------------
-----

* Free software: Apache license
* Documentation: https://rally.readthedocs.org/en/latest/
* Source: https://git.openstack.org/cgit/openstack/rally
* Bugs: https://bugs.launchpad.net/rally
* Step-by-step tutorial: https://rally.readthedocs.io/en/latest/quick_start/tutorial.html
* RoadMap: https://docs.google.com/a/mirantis.com/spreadsheets/d/16DXpfbqvlzMFaqaXAcJsBzzpowb_XpymaK2aFY2gA2g
* Launchpad page: https://launchpad.net/rally
* Gitter chat: https://gitter.im/rally-dev/Lobby
* Trello board: https://trello.com/b/DoD8aeZy/rally
1 change: 1 addition & 0 deletions devstack/lib/rally
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ fi
# install_rally() - Collect source and prepare
function install_rally() {
setup_develop $RALLY_DIR
sudo pip install rally_openstack>=1.1.0
}

# configure_rally() - Set config files, create data dirs, etc
Expand Down
19 changes: 19 additions & 0 deletions doc/release_notes/archive/v1.0.0.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
============
Rally v1.0.0
============

+------------------+-----------------------+
| Release date | **06/20/2018** |
+------------------+-----------------------+

.. warning:: There is no in-tree OpenStack plugins anymore. They moved to
`the separate project <https://github.com/openstack/rally-openstack>`_.

Starting from the current release all notes are written in a single file
`CHANGELOG.rst <https://github.com/openstack/rally/tree/master/CHANGELOG.rst>`_
Follow it for more details.

Thanks
~~~~~~

2 Everybody!
2 changes: 1 addition & 1 deletion doc/release_notes/latest.rst
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ and textual information.
max: 0


hook_example_script.sh generates dummy output in JSON format. Grafical
hook_example_script.sh generates dummy output in JSON format. Graphical
information format is the same as for workloads and the same types of
charts are supported for the hooks.

Expand Down
2 changes: 1 addition & 1 deletion doc/source/task/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -366,7 +366,7 @@ CLI References
==============

For more information regarding Rally Task Component CLI please proceed
to `CLI reference <../cli/cli_reference.html#category-task>`_
to `CLI reference <../cli_reference.html#category-task>`_

.. references:

Expand Down
3 changes: 2 additions & 1 deletion doc/source/verification/overview.rst
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,8 @@ for:
failed tests from it and so on. Btw, all verification runs arguments are
stored in the database.
* **Processing results**. Rally DataBase stores all `verifications
<link-to-glossary>`_ and you can obtain unified (across different verifiers)
<../overview/glossary.html#verification>`_
and you can obtain unified (across different verifiers)
results at any time. You can find a verification run summary there, run
arguments which were used, error messages and etc. Comparison mechanism for
several verifications is available too. Verification reports can be generated
Expand Down
4 changes: 2 additions & 2 deletions doc/specs/implemented/deployment_type.rst
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ unbound it from OpenStack.
Problem description
===================

Rally is able to examine only system that use Keystone as a authentication
services, which limits sphere where Rally is suitable.
Rally is able to examine only system that use Keystone as an
authentication service, which limits sphere where Rally is suitable.

At the moment to run Rally Task or Rally Verify you must specify OpenStack
deployment which contains credentials of it. These credentials are used in
Expand Down
2 changes: 1 addition & 1 deletion doc/specs/in-progress/cleanup_refactoring.rst
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ Alternatives
the tenants, regardless of resource naming, so we only need to keep track of
Rally tenants (naming could be a solution here) and resources in admin
tenant. In this case, we need to think about a case where Rally needs to
cleanup some resources from a existing tenant while leaving the rest
cleanup some resources from an existing tenant while leaving the rest
available.

* Use/enhance Tempest cleanup command (tempest/cmd/cleanup.py). Compare
Expand Down
4 changes: 2 additions & 2 deletions etc/rally.bash_completion
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ _rally()
OPTS["plugin_show"]="--name --platform"
OPTS["task_abort"]="--uuid --soft"
OPTS["task_delete"]="--force --uuid"
OPTS["task_detailed"]="--uuid --iterations-data"
OPTS["task_detailed"]="--uuid --iterations-data --filter-by"
OPTS["task_export"]="--uuid --type --to"
OPTS["task_import"]="--file --deployment --tag"
OPTS["task_list"]="--deployment --all-deployments --status --tag --uuids-only"
Expand Down Expand Up @@ -105,4 +105,4 @@ _rally()
return 0
}

complete -o filenames -F _rally rally
complete -o filenames -F _rally rally
Loading