path: root/regtest
AgeCommit message (Collapse)AuthorFilesLines
2018-11-18regtest: Fix file opened in wrong mode.Elliott Sales de Andrade1-3/+2
2018-11-18regtest: Fix Printer.printerr.Elliott Sales de Andrade1-2/+2
2018-11-18regtest: Replace execfile with plain read & exec.Elliott Sales de Andrade1-1/+2
2018-11-18regtest: Enable new-style division everywhere.Elliott Sales de Andrade22-25/+29
2018-11-18regtest: Enable absolute import on all files.Elliott Sales de Andrade22-22/+22
2018-11-18regtest: Replace tabs with spaces.Elliott Sales de Andrade1-1/+1
2018-11-18regtest: Use print function everywhere.Elliott Sales de Andrade22-1/+23
2018-11-18regtest: Fix iteration through dictionary on Py3.Elliott Sales de Andrade2-7/+2
2018-11-18regtest: Don't use exceptions for Printer singleton.Elliott Sales de Andrade1-14/+12
In Python 3, exceptions must derive from the BaseException class, which Printer definitely does not do.
2017-12-21regtest: Add an option to exit after n failuresCarlos Garcia Campos4-3/+68
Note that running jobs are not cancelled when max failures is reached, so we usually end up getting more failures than the maximum.
2017-09-24regtest: change default value of utils dir to ../build/utilsCarlos Garcia Campos1-1/+1
2016-01-02regtest: Fix the number of worker threads spawned reported in log of ↵Carlos Garcia Campos1-1/+1
run-tests command
2016-01-02regtest: Limit the number of worker threads to the number of documents to ↵Adam Reichold1-9/+18
create references The same way we do for the run-tests command.
2016-01-01regtest: Allow to interrupt run-tests and create-refs commands when multiple ↵Carlos Garcia Campos3-4/+72
threads are used The Queue join implementation uses a non-timed wait that blocks the main thread, making it impossible to interrupt it with CTRL+C or sending SIGINT signal. Using any timeout value for wait would fix the problem, but Queue doesn't allow to pass a timeout to the join method. The Queue implementation is actually quite simple, so we can just add our own implementation with only the things we really need and use a timeout value when calling wait() in join().
2016-01-01regtest: Do not use the log printer with the TestReferences lock heldAdam Reichold1-1/+1
2015-12-04regtest: Pass always both the owner and user passwords to the toolsCarlos Garcia Campos4-4/+4
2015-12-03regtest: Fix testing text backendCarlos Garcia Campos1-1/+1
Fix a typo in previous commit.
2015-12-03regtest: Add support for testing password protected documentsCarlos Garcia Campos9-15/+49
Similar to how skipped file works, you can pass a passwords file from the command line or add a Passwords file to your docs directory. This file should be a python file containing a "passwords" dictionary where the key is the test document and the value is the password required to open that document.
2015-06-04regtest: Disable pretty diffs for text files in HTML reportsCarlos Garcia Campos1-2/+5
difflib is entering in an infinite loop with some files, so disable it for now. We need to either fix that somehow or find a different way to generate pretty diffs of text files.
2014-11-07regtest: Update references also for tests that no longer crashCarlos Garcia Campos2-0/+26
We are currently updating the refs only when we have checksums to compare, but if a test used to crash there's no md5 file in refs.
2014-10-30regtest: Try to fix again the paths of the resources in the generated HTML ↵Carlos Garcia Campos1-24/+23
report Use always relatives paths for the resources under the out directory, and absolute paths for all other resources when no-absolute-path command line option is used.
2014-10-29regtest: respect --no-absolute-paths when creating the pretty diff linksCarlos Garcia Campos1-6/+10
2014-10-27regtest: Add an option to create the HTML report without absolute pathsCarlos Garcia Campos2-4/+15
It uses the paths as received from the command line attributes.
2014-10-26regtest: Fix png diff generation with python-pil pillow forkCarlos Garcia Campos1-1/+1
Use from PIL import Image instead of just import Image, since pillow only doesn't support import Image without using PIL.Image.
2014-10-26regtest: Limit the number of worker threads to the number of documents to testCarlos Garcia Campos1-9/+18
We are always spawning all the threads even if the documents to test is less than the worker threads. Also optimize the case of running only one test to not spwn any thread.
2014-10-26regtest: Allow to run groups of tests individuallyCarlos Garcia Campos3-13/+49
Now it's possible to pass more than one argument to run-tests command and optionally the docs directory. When more than one test is passed and the docs directory is not provided, the common base path of all passed tests is used as docs directory. The tests passed can be documents or directories, using absolute paths or paths relative to the docs directory. This also allows us to update the refs for a group of tests.
2013-12-15regtest: Add a command line option to create-report command to not launch ↵Carlos Garcia Campos2-3/+7
the browser
2013-12-15regtest: Return an exist status code depending on whether the command succeededCarlos Garcia Campos8-10/+20
2013-12-06regtest: Limit the stderr files to ~1MBCarlos Garcia Campos1-1/+5
Some tests send a lot of information to stderr, usually due to parsing errors in buggy documents. More than 1MB of stderr output is diffcult to hanlde and in most cases it's redundant with a lot of duplicated messages. This patch reduced the size of the refs dir for the complete test suite by 1GB.
2013-12-06regtest: Save checksum results sorted in md5 filesCarlos Garcia Campos1-1/+1
We are using os.listdir() to get the list of test results that returns files in arbitrary order.
2013-12-06regtest: Show also the tests expected to crash/fail to run but don't failCarlos Garcia Campos1-1/+7
2013-12-06regtest: Improve readability of test resultsCarlos Garcia Campos1-20/+47
Show a summary of tests failed per backend with the percentages and use a new line for every test in the result instead of using a comma separated line.
2013-11-29regtest: Do not buffer stderr outputCarlos Garcia Campos1-10/+30
Some buggy documents can produce a huge stderr output because of parsing errors or whatever. We could give a file directly to Popen to write the stderr file, but we only want to create the file when there's output, because it's what we use to know whether the command produced output or not. So, instead of buffering the whole output and then write it to the file, now we read from the pipe while the command is running, writing the output in chunks to the file. This improves a lot the memory consumption when running some tests.
2013-11-29regtest: Read test results in chunks to get the md5 digestCarlos Garcia Campos1-7/+12
Some backends can generate huge results, like huge postscript files that we don't want to load in memory to get the md5. So, instead of creating thr md5 object with the entire file, we feed it with chunks of data using the update method. This improves a lot the memory consumption and performance as well.
2013-11-29regtest: Remove unused method _check_exit_status2Carlos Garcia Campos1-20/+0
It was used when the backends ran in parallel odd and even pages, but it's no longer used since threads support was added.
2013-11-29regtest: Do not store the current line in Printer but only its lengthCarlos Garcia Campos1-5/+5
We are not using the line text anymore, but only the length.
2013-11-27regtest: Do not consider docs with no refs as skippedCarlos Garcia Campos1-17/+23
Handle them differently as new docs, and show them in the summary suggesting to use create-refs command to include them in the test suite.
2013-11-27regtest: Use number of tests run to show the progressCarlos Garcia Campos3-112/+118
Instead of using the number of document tested. We don't really need to set a number to a particular document, we only want to know the progress of the whole process. This ensures that the progress is shown in the correct order. It also simplifies the code a bit. Also improved the output formatting to make it easier to read.
2012-12-29regtest: Print test results when the test has finishedCarlos Garcia Campos2-15/+28
To make sure the result corresponds to the test now that we are using multiple threads.
2012-12-08regtest: Use the number of cpus as default number of worker threadsCarlos Garcia Campos1-3/+5
2012-12-08regtest: Remove trailing whitespacesCarlos Garcia Campos3-13/+13
2012-12-06implement parallel testing using Python's Queue classAdam Reichold6-34/+104
2012-10-21regstest: Add print_test_result_ln() and use it for failed test resultsCarlos Garcia Campos2-5/+8
To make sure they are always shown.
2012-10-21regtest: Do not rewrite lines in verbose modeCarlos Garcia Campos1-1/+1
2012-09-23regtest: Reduce the noise of the default output when running testsCarlos Garcia Campos8-29/+137
Show permanent information only about failed tests, without the details about the failing pages. Previous verbose output is available passing --verbose command line output.
2012-04-01regtest: remove debug print to show bisect exit status in Bisect.pyCarlos Garcia Campos1-1/+0
2012-04-01regtest: Fix crash when creating HTML report for a single testCarlos Garcia Campos1-4/+1
2012-04-01regtest: Don't show results for crashed or failed to run tests in HTML reportCarlos Garcia Campos1-0/+2
2012-03-11regtest: Add find-regression command to run git bisect automaticallyCarlos Garcia Campos4-0/+339
2012-03-10regtest: Add information about skipped tests in the summaryCarlos Garcia Campos1-3/+9