saundersp/ViolaJones
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Compare commits

base: saundersp:8565ce782bd86aea3d89cf736cf656d08fe7450c
Branches Tags
saundersp:main
..
compare: saundersp:main
Branches Tags
saundersp:main

10 Commits
8565ce782b ... main

Author SHA1 Message Date
saundersp
b1abaed9e3 README.fr.md : Fix typos 2025-04-30 21:33:52 +02:00
saundersp
2051ae8cdc downloader/convert_dataset.py : Added better typing and formatting 2024-11-08 01:23:38 +01:00
saundersp
e9df962d7a Changed VENV_PATH from 'venv' to '.venv' 2024-11-08 01:22:49 +01:00
saundersp
512a21cdb3 Updated dependencies 2024-11-08 01:18:31 +01:00
saundersp
8c7f2c1c97 python/Makefile : fix test target 2024-07-22 22:07:17 +02:00
saundersp
b55713cae3 Fix some typos 2024-07-22 22:06:51 +02:00
saundersp
d83ddd04c1 docker-compose.yaml : disabled pulling for custom images 2024-07-22 22:04:49 +02:00
saundersp
294eb7fa5b Updated dependencies 2024-07-22 22:03:59 +02:00
saundersp
95eaffda4d Added testing targets 2024-07-22 22:03:30 +02:00
saundersp
1ec28f78de Updated Dockerfiles 2024-06-28 20:04:33 +02:00
19 changed files with 227 additions and 163 deletions
Expand all files Collapse all files

2
.gitignore vendored
View File

@ -1,6 +1,6 @@
data data
*/models* */models*
venv .venv
*/out* */out*
python/__pycache__ python/__pycache__
cpp/bin cpp/bin

10
README.fr.md
View File

@ -1,6 +1,6 @@
# Viola Jones # Viola Jones
*Lisez ceci dans d'autres langues: [English](README.md)* *Lisez ceci dans d'autres langues : [English](README.md)*
## Description ## Description
@ -23,7 +23,7 @@ Implémentation de l'algorithme "Viola Jones" en Python et C++.
Vous pouvez configurer l'algorithme avec les variables globales définies au début du fichier *ViolaJones.cpp* puis lancer 'make start'. Vous pouvez configurer l'algorithme avec les variables globales définies au début du fichier *ViolaJones.cpp* puis lancer 'make start'.
Il y a également la commande 'make clean' qui permet de supprimer tout fichiers compilées. Il y a également la commande 'make clean' qui permet de supprimer tous fichiers compilés.
### Python ### Python
@ -34,7 +34,7 @@ Vous pouvez configurer l'algorithme dans le fichier *config.py* puis lancer l'al
## Entraînement ## Entraînement
L'algorithme à été entraîné avec un processeur Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz et un GPU NVIDIA GeForce RTX 2080 Ti. L'algorithme a été entraîné avec un processeur Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz et un GPU NVIDIA GeForce RTX 2080 Ti.
### Tableau de comparaison des temps d'exécution ### Tableau de comparaison des temps d'exécution
@ -66,7 +66,7 @@ Il se trouve que le GPU bat systématiquement le CPU en matière de temps d'exé
L'algorithme de ViolaJones étant déterministe, tous les modèles entraînés avec un T donnée, peu importe le moyen (CPU, NJIT ou GPU), seront les mêmes modèles avec les mêmes paramètres. L'algorithme de ViolaJones étant déterministe, tous les modèles entraînés avec un T donnée, peu importe le moyen (CPU, NJIT ou GPU), seront les mêmes modèles avec les mêmes paramètres.
Rappel: ACC (Accuracy i.e. Précision), F1 (Score F1), FN (Faux Négatif) et FP (Faux Positif). Rappel : ACC (Accuracy i.e. Précision), F1 (Score F1), FN (Faux Négatif) et FP (Faux Positif).
| Evaluating | ACC (E) | F1 (E) | FN (E) | FP (E) | ACC (T) | F1 (T) | FN (T) | FP (T) | | Evaluating | ACC (E) | F1 (E) | FN (E) | FP (E) | ACC (T) | F1 (T) | FN (T) | FP (T) |
| ------------------ | ------- | ------ | ------ | ------ | ------- | ------ | ------ | ------ | | ------------------ | ------- | ------ | ------ | ------ | ------- | ------ | ------ | ------ |
@ -202,7 +202,7 @@ L'algorithme de ViolaJones étant déterministe, les fichiers devraient être é
| ViolaJones T = 200 (NJIT) | 3,989,600 | 3ms 989µs 600ns | 15,957,700 | 15ms 957µs 700ns | | ViolaJones T = 200 (NJIT) | 3,989,600 | 3ms 989µs 600ns | 15,957,700 | 15ms 957µs 700ns |
| ViolaJones T = 300 (NJIT) | 5,983,900 | 5ms 983µs 900ns | 23,935,500 | 23ms 935µs 500ns | | ViolaJones T = 300 (NJIT) | 5,983,900 | 5ms 983µs 900ns | 23,935,500 | 23ms 935µs 500ns |
## Resources additionnels ## Ressources additionnelles
- [Rapid Object Detection using a Boosted Cascade of Simple Features](https://www.cs.cmu.edu/~efros/courses/LBMV07/Papers/viola-cvpr-01.pdf) - [Rapid Object Detection using a Boosted Cascade of Simple Features](https://www.cs.cmu.edu/~efros/courses/LBMV07/Papers/viola-cvpr-01.pdf)
- [Chapter 39. Parallel Prefix Sum (Scan) with CUDA](https://developer.nvidia.com/gpugems/gpugems3/part-vi-gpu-computing/chapter-39-parallel-prefix-sum-scan-cuda) - [Chapter 39. Parallel Prefix Sum (Scan) with CUDA](https://developer.nvidia.com/gpugems/gpugems3/part-vi-gpu-computing/chapter-39-parallel-prefix-sum-scan-cuda)

6
cpp/Dockerfile
View File

@ -1,11 +1,11 @@
FROM nvidia/cuda:12.5.0-devel-ubi9 as builder FROM nvidia/cuda:12.6.2-devel-ubi9 AS builder
WORKDIR /home/ViolaJones/cpp WORKDIR /home/ViolaJones/cpp
COPY *.cu *.cpp *.hpp Makefile ./ COPY *.cu *.cpp *.hpp Makefile ./
RUN make -j "$(nproc)" RUN make -j "$(nproc)" && make -j "$(nproc)" ./bin/ViolaJonesTest
FROM nvidia/cuda:12.5.0-base-ubi9 FROM nvidia/cuda:12.6.2-base-ubi9
WORKDIR /home/ViolaJones/cpp WORKDIR /home/ViolaJones/cpp

16
cpp/Makefile
View File

@ -10,8 +10,10 @@ CFLAGS := -dlto -O2 -Xcompiler -O2
#CFLAGS := -dlto -O2 -g -Xcompiler -O2,-g,-ggdb #CFLAGS := -dlto -O2 -g -Xcompiler -O2,-g,-ggdb
CFLAGS := $(CFLAGS) -MMD -MP -Werror=all-warnings -Xcompiler -Wall,-Werror,-Wextra CFLAGS := $(CFLAGS) -MMD -MP -Werror=all-warnings -Xcompiler -Wall,-Werror,-Wextra
EXEC := $(OBJ_DIR)/ViolaJones EXEC := $(OBJ_DIR)/ViolaJones
EXEC_TEST := $(OBJ_DIR)/ViolaJonesTest
DATA := $(DATA_PATH)/X_train.bin $(DATA_PATH)/X_test.bin $(DATA_PATH)/y_train.bin $(DATA_PATH)/y_test.bin DATA := $(DATA_PATH)/X_train.bin $(DATA_PATH)/X_test.bin $(DATA_PATH)/y_train.bin $(DATA_PATH)/y_test.bin
SRC := $(shell find $(SRC_DIR) -name '*.cpp' -o -name '*.cu') SRC := $(shell find $(SRC_DIR) \( -name '*.cpp' -o -name '*.cu' \) -and -not -name projet_test.cpp)
SRC_TEST := $(shell find $(SRC_DIR) \( -name '*.cpp' -o -name '*.cu' \) -and -not -name projet.cpp)
OBJ_EXT := o OBJ_EXT := o
ifeq ($(OS), Windows_NT) ifeq ($(OS), Windows_NT)
EXEC := $(EXEC).exe EXEC := $(EXEC).exe
@ -19,6 +21,8 @@ ifeq ($(OS), Windows_NT)
endif endif
OBJ := $(SRC:$(SRC_DIR)/%.cpp=$(OBJ_DIR)/%.$(OBJ_EXT)) OBJ := $(SRC:$(SRC_DIR)/%.cpp=$(OBJ_DIR)/%.$(OBJ_EXT))
OBJ := $(OBJ:$(SRC_DIR)/%.cu=$(OBJ_DIR)/%.$(OBJ_EXT)) OBJ := $(OBJ:$(SRC_DIR)/%.cu=$(OBJ_DIR)/%.$(OBJ_EXT))
OBJ_TEST := $(SRC_TEST:$(SRC_DIR)/%.cpp=$(OBJ_DIR)/%.$(OBJ_EXT))
OBJ_TEST := $(OBJ_TEST:$(SRC_DIR)/%.cu=$(OBJ_DIR)/%.$(OBJ_EXT))
.PHONY: all .PHONY: all
all: $(EXEC) all: $(EXEC)
@ -42,6 +46,12 @@ $(EXEC): $(OBJ)
@echo Linking objects files to $@ @echo Linking objects files to $@
@$(CC) $(CFLAGS) $^ -o $@ @$(CC) $(CFLAGS) $^ -o $@
# FIXME When using the docker image, Make check prequisites even when the target already exists
#$(EXEC_TEST): $(OBJ_TEST) | check-nvcc-works
$(EXEC_TEST): $(OBJ_TEST)
@echo Linking objects files to $@
@$(CC) $(CFLAGS) $^ -o $@
$(DATA): $(DATA):
@echo 'Missing $(DATA) files, use downloader first' && exit 1 @echo 'Missing $(DATA) files, use downloader first' && exit 1
@ -49,6 +59,10 @@ $(DATA):
start: $(EXEC) $(DATA) start: $(EXEC) $(DATA)
@./$(EXEC) @./$(EXEC)
.PHONY: test
test: $(EXEC_TEST)
@./$(EXEC_TEST)
.PHONY: debug .PHONY: debug
debug: $(EXEC) $(DATA) debug: $(EXEC) $(DATA)
#@cuda-gdb -q $(EXEC) #@cuda-gdb -q $(EXEC)

6
cpp/ViolaJones.hpp
View File

@ -9,7 +9,7 @@
* @tparam T Inner type of the arrays to test * @tparam T Inner type of the arrays to test
* @param cpu CPU Array * @param cpu CPU Array
* @param gpu GPU Array * @param gpu GPU Array
* @return Whether the test was succesful * @return Whether the test was successful
*/ */
template <typename T> template <typename T>
bool unit_test_cpu_vs_gpu(const np::Array<T>& cpu, const np::Array<T>& gpu) noexcept { bool unit_test_cpu_vs_gpu(const np::Array<T>& cpu, const np::Array<T>& gpu) noexcept {
@ -249,7 +249,7 @@ np::Array<uint8_t> classify_viola_jones(const np::Array<float64_t>&, const np::A
np::Array<float64_t> init_weights(const np::Array<uint8_t>&) noexcept; np::Array<float64_t> init_weights(const np::Array<uint8_t>&) noexcept;
/** /**
* @brief Select the best classifer given their predictions. * @brief Select the best classifier given their predictions.
* *
* @param classifiers The weak classifiers * @param classifiers The weak classifiers
* @param weights Trained weights of each classifiers * @param weights Trained weights of each classifiers
@ -261,7 +261,7 @@ std::tuple<int32_t, float64_t, np::Array<float64_t>> select_best(const np::Array
const np::Array<uint8_t>&) noexcept; const np::Array<uint8_t>&) noexcept;
/** /**
* @brief Train the weak calssifiers. * @brief Train the weak classifiers.
* *
* @param T Number of weak classifiers * @param T Number of weak classifiers
* @param X_feat Integrated features * @param X_feat Integrated features

2
cpp/docker-compose.yaml
View File

@ -1,6 +1,7 @@
services: services:
violajones-cpp: violajones-cpp:
image: saundersp/violajones-cpp image: saundersp/violajones-cpp
pull_policy: never
build: . build: .
volumes: volumes:
- ./models:/home/ViolaJones/cpp/models - ./models:/home/ViolaJones/cpp/models
@ -11,5 +12,4 @@ services:
reservations: reservations:
devices: devices:
- driver: nvidia - driver: nvidia
count: 1
capabilities: [gpu] capabilities: [gpu]

31
cpp/projet_test.cpp Normal file
View File

@ -0,0 +1,31 @@
#include "toolbox.hpp"
#include "config.hpp"
#include "toolbox_unit_test.hpp"
#include "ViolaJones.hpp"
#if GPU_BOOSTED
#include "gpu_unit_test.hpp"
#endif
int32_t main(void){
setlocale(LC_NUMERIC, ""); // Allow proper number display
const std::chrono::system_clock::time_point unit_timestamp = perf_counter_ns();
const std::array<int32_t, 3> unit_gaps = { 27, -18, 29 };
header(unit_gaps, { "Unit testing", "Time spent (ns)", "Formatted time spent" });
#if GPU_BOOSTED
benchmark_function_void("Testing GPU capabilities 1D", unit_gaps[0], test_working, 50000);
benchmark_function_void("Testing GPU capabilities 2D", unit_gaps[0], test_working_2d, 200, 500);
benchmark_function_void("Testing GPU capabilities 3D", unit_gaps[0], test_working_3d, 30, 40, 500);
#endif
benchmark_function_void("Testing format_time", unit_gaps[0], format_time_test);
benchmark_function_void("Testing format_time_ns", unit_gaps[0], format_time_ns_test);
benchmark_function_void("Testing format_byte_size", unit_gaps[0], format_byte_size_test);
benchmark_function_void("Testing thousand_sep", unit_gaps[0], thousand_sep_test);
const long long time_spent = duration_ns(perf_counter_ns() - unit_timestamp);
formatted_line(unit_gaps, "├", "┼", "─", "┤");
formatted_row(unit_gaps, { "Unit testing summary", thousand_sep(time_spent).c_str(), format_time_ns(time_spent).c_str() });
footer(unit_gaps);
return EXIT_SUCCESS;
}

4
downloader/Dockerfile
View File

@ -1,6 +1,6 @@
FROM alpine:3.20.0 FROM alpine:3.20.3
RUN apk add --no-cache curl=8.7.1-r0 python3=3.12.3-r1 && rm -rf /var/cache/apk* RUN apk add --no-cache curl=8.11.0-r1 python3=3.12.7-r0 && rm -rf /var/cache/apk*
WORKDIR /home/ViolaJones/downloader WORKDIR /home/ViolaJones/downloader
COPY requirements.txt activate.sh ./ COPY requirements.txt activate.sh ./

6
downloader/activate.sh
View File

@ -4,16 +4,14 @@
set -e set -e
test -z "$EXEC_DIR" && EXEC_DIR=. test -z "$EXEC_DIR" && EXEC_DIR=.
test -z "$VENV_PATH" && VENV_PATH="$EXEC_DIR/venv" test -z "$VENV_PATH" && VENV_PATH="$EXEC_DIR/.venv"
activate(){ activate(){
if [ ! -d "$VENV_PATH" ]; then if [ ! -d "$VENV_PATH" ]; then
echo 'Creating python virtual environnement' echo 'Creating python virtual environnement'
python -m venv "$VENV_PATH" python -m venv --upgrade-deps "$VENV_PATH"
echo 'Activating virtual environnement' echo 'Activating virtual environnement'
activate activate
echo 'Updating base pip packages'
python -m pip install -U setuptools pip
echo 'Installing requirements' echo 'Installing requirements'
pip install -r requirements.txt pip install -r requirements.txt
elif [ -f "$VENV_PATH"/Scripts/activate ]; then . "$VENV_PATH"/Scripts/activate elif [ -f "$VENV_PATH"/Scripts/activate ]; then . "$VENV_PATH"/Scripts/activate

48
downloader/convert_dataset.py
View File

@ -3,29 +3,31 @@ from tqdm import tqdm
from functools import partial from functools import partial
from sys import argv from sys import argv
import numpy as np import numpy as np
from numpy.typing import NDArray
from typing import Final, Callable
from os import path, listdir from os import path, listdir
# Induce determinism # Induce determinism
np.random.seed(133742) np.random.seed(196_863)
# Makes the "leave" argument default to False # Makes the 'leave' argument default to False
tqdm = partial(tqdm, leave = False) tqdm: Callable = partial(tqdm, leave = False)
def read_pgm(pgm_file: BufferedReader) -> np.ndarray: def read_pgm(pgm_file: BufferedReader) -> NDArray[np.uint8]:
"""Read the data of a PGM file """Read the data of a PGM file
Args: Args:
pgm_file (BufferedReader): PGM File pgm_file (BufferedReader): PGM File
Returns: Returns:
np.ndarray: PGM data NDArray[np.uint8]: PGM data
""" """
assert (f := pgm_file.readline()) == b'P5\n', f"Incorrect file format: {f}" assert (f := pgm_file.readline()) == b'P5\n', f'Incorrect file format: {f}'
(width, height) = [int(i) for i in pgm_file.readline().split()] (width, height) = (int(i) for i in pgm_file.readline().split())
assert width > 0 and height > 0, f"Incorrect dimensions: {width}x{height}" assert width > 0 and height > 0, f'Incorrect dimensions: {width}x{height}'
assert (depth := int(pgm_file.readline())) < 256, f"Incorrect depth: {depth}" assert (depth := int(pgm_file.readline())) < 256, f'Incorrect depth: {depth}'
buff = np.empty(height * width, dtype = np.uint8) buff: Final[NDArray[np.uint8]] = np.empty(height * width, dtype = np.uint8)
for i in range(buff.shape[0]): for i in range(buff.shape[0]):
buff[i] = ord(pgm_file.read(1)) buff[i] = ord(pgm_file.read(1))
return buff.reshape((height, width)) return buff.reshape((height, width))
@ -36,25 +38,25 @@ def __main__(data_path: str) -> None:
Args: Args:
data_path (str): Path of the PGM files data_path (str): Path of the PGM files
""" """
for set_name in tqdm(["train", "test"], desc = "set name"): for set_name in tqdm(['train', 'test'], desc = 'set name'):
X, y = [], [] X, y = [], []
for y_i, label in enumerate(tqdm(["non-face", "face"], desc = "label")): for y_i, label in enumerate(tqdm(['non-face', 'face'], desc = 'label')):
for filename in tqdm(listdir(f"{data_path}/{set_name}/{label}"), desc = "Reading pgm file"): for filename in tqdm(listdir(f'{data_path}/{set_name}/{label}'), desc = 'Reading pgm file'):
with open(f"{data_path}/{set_name}/{label}/{filename}", "rb") as face: with open(f'{data_path}/{set_name}/{label}/{filename}', 'rb') as face:
X.append(read_pgm(face)) X.append(read_pgm(face))
y.append(y_i) y.append(y_i)
X, y = np.asarray(X), np.asarray(y) X, y = np.asarray(X), np.asarray(y)
idx = np.random.permutation(y.shape[0]) idx: NDArray[np.int64] = np.random.permutation(y.shape[0])
X, y = X[idx], y[idx] X, y = X[idx], y[idx]
for org, s in tqdm(zip("Xy", [X, y]), desc = f"Writing {set_name}"): for org, s in tqdm(zip('Xy', [X, y]), desc = f'Writing {set_name}'):
with open(f"{data_path}/{org}_{set_name}.bin", "w") as out: with open(f'{data_path}/{org}_{set_name}.bin', 'w') as out:
out.write(f'{str(s.shape)[1:-1].replace(",", "")}\n') out.write(f'{str(s.shape)[1:-1].replace(',', '')}\n')
raw = s.ravel() raw: NDArray = s.ravel()
for s_i in tqdm(raw[:-1], desc = f"Writing {org}"): for s_i in tqdm(raw[:-1], desc = f'Writing {org}'):
out.write(f"{s_i} ") out.write(f'{s_i} ')
out.write(str(raw[-1])) out.write(str(raw[-1]))
if __name__ == "__main__": if __name__ == '__main__':
__main__(argv[1]) if len(argv) == 2 else print(f"Usage: python {__file__[__file__.rfind(path.sep) + 1:]} ./data_location") __main__(argv[1]) if len(argv) == 2 else print(f'Usage: python {__file__[__file__.rfind(path.sep) + 1:]} ./data_location')

1
downloader/docker-compose.yaml
View File

@ -1,6 +1,7 @@
services: services:
downloader: downloader:
image: saundersp/violajones-downloader image: saundersp/violajones-downloader
pull_policy: never
build: . build: .
volumes: volumes:
- ../data:/home/ViolaJones/data - ../data:/home/ViolaJones/data

4
downloader/requirements.txt
View File

@ -1,2 +1,2 @@
numpy==1.26.4 numpy==2.1.3
tqdm==4.66.2 tqdm==4.67.0

6
python/Dockerfile
View File

@ -1,8 +1,8 @@
FROM nvidia/cuda:12.5.0-devel-ubi9 as builder FROM nvidia/cuda:12.6.2-devel-ubi9
RUN dnf install -y python3.11-3.11.7-1.el9 \ RUN dnf install -y python3.12-3.12.1-4.el9_4.4 \
&& dnf clean all \ && dnf clean all \
&& ln -s /usr/bin/python3 /usr/bin/python && ln -s /usr/bin/python3.12 /usr/bin/python
WORKDIR /home/ViolaJones/python WORKDIR /home/ViolaJones/python
COPY Makefile activate.sh requirements.txt ./ COPY Makefile activate.sh requirements.txt ./

10
python/Makefile
View File

@ -17,6 +17,10 @@ venv:
start: $(DATA) | venv check-python-works start: $(DATA) | venv check-python-works
@sh -c '. ./activate.sh && python projet.py' @sh -c '. ./activate.sh && python projet.py'
.PHONY: test
test: | venv check-python-works
@sh -c '. ./activate.sh && python project_test.py'
.PHONY: debug .PHONY: debug
debug: $(DATA) | venv check-python-works check-pudb-works debug: $(DATA) | venv check-python-works check-pudb-works
@bash -c 'source activate.sh && pudb projet.py' @bash -c 'source activate.sh && pudb projet.py'
@ -45,9 +49,7 @@ log: $(DATA) reset | venv
.PHONY: reset .PHONY: reset
reset: reset:
@echo 'Deleting generated states and models' @echo 'Deleting generated states and models'
@rm -frv $(OUT_DIR)/* $(MODELS_DIR)/* @rm -frv $(OUT_DIR) $(MODELS_DIR)
#@ln -sv /mnt/pierre_stuffs/ViolaJones/python/models .
#@ln -sv /mnt/pierre_stuffs/ViolaJones/python/out .
.PHONY: clean .PHONY: clean
clean: clean:
@ -55,7 +57,7 @@ clean:
.PHONY: mrproper .PHONY: mrproper
mrproper: clean mrproper: clean
@rm -rfv __pycache__ venv @rm -rfv __pycache__ .venv
.PHONY: help .PHONY: help
help: help:

6
python/activate.sh
View File

@ -4,16 +4,14 @@
set -e set -e
test -z "$EXEC_DIR" && EXEC_DIR=. test -z "$EXEC_DIR" && EXEC_DIR=.
test -z "$VENV_PATH" && VENV_PATH="$EXEC_DIR/venv" test -z "$VENV_PATH" && VENV_PATH="$EXEC_DIR/.venv"
activate(){ activate(){
if [ ! -d "$VENV_PATH" ]; then if [ ! -d "$VENV_PATH" ]; then
echo 'Creating python virtual environnement' echo 'Creating python virtual environnement'
python -m venv "$VENV_PATH" python -m venv --upgrade-deps "$VENV_PATH"
echo 'Activating virtual environnement' echo 'Activating virtual environnement'
activate activate
echo 'Updating base pip packages'
python -m pip install -U setuptools pip
echo 'Installing requirements' echo 'Installing requirements'
pip install -r requirements.txt pip install -r requirements.txt
elif [ -f "$VENV_PATH"/Scripts/activate ]; then . "$VENV_PATH"/Scripts/activate elif [ -f "$VENV_PATH"/Scripts/activate ]; then . "$VENV_PATH"/Scripts/activate

2
python/docker-compose.yaml
View File

@ -1,6 +1,7 @@
services: services:
violajones-python: violajones-python:
image: saundersp/violajones-python image: saundersp/violajones-python
pull_policy: never
build: . build: .
volumes: volumes:
- ./models:/home/ViolaJones/python/models - ./models:/home/ViolaJones/python/models
@ -11,5 +12,4 @@ services:
reservations: reservations:
devices: devices:
- driver: nvidia - driver: nvidia
count: 1
capabilities: [gpu] capabilities: [gpu]

15
python/project_test.py Normal file
View File

@ -0,0 +1,15 @@
from toolbox_unit_test import format_time_test, format_time_ns_test
from toolbox import header, footer, formatted_row, formatted_line
from toolbox import format_time_ns, benchmark_function
from time import perf_counter_ns
if __name__ == '__main__':
unit_timestamp = perf_counter_ns()
unit_gaps = [27, -18, 29]
header(unit_gaps, ['Unit testing', 'Time spent (ns)', 'Formatted time spent'])
benchmark_function('testing format_time', unit_gaps[0], format_time_test)
benchmark_function('testing format_time_ns', unit_gaps[0], format_time_ns_test)
time_spent = perf_counter_ns() - unit_timestamp
formatted_line(unit_gaps, '├', '┼', '─', '┤')
formatted_row(unit_gaps, ['Unit testing summary', f'{time_spent:,}', format_time_ns(time_spent)])
footer(unit_gaps)

8
python/requirements.txt
View File

@ -1,3 +1,5 @@
numba==0.59.1 numba==0.60.0
scikit-learn==1.4.1.post1 scikit-learn==1.5.2
tqdm==4.66.2 tqdm==4.67.0
#pudb==2024.1.3

207
python/toolbox_unit_test.py
View File

@ -1,5 +1,5 @@
from typing import Any
from toolbox import format_time, format_time_ns from toolbox import format_time, format_time_ns
from typing import Any
def Assert(name: str, expected: Any, result: Any): def Assert(name: str, expected: Any, result: Any):
"""Test if a given result is equal of the expected one and log result """Test if a given result is equal of the expected one and log result
@ -18,115 +18,116 @@ def format_time_test() -> None:
See https://en.wikipedia.org/wiki/Unit_of_time for details See https://en.wikipedia.org/wiki/Unit_of_time for details
""" """
Assert("format_time null", "0s", format_time(0)) Assert('format_time null', '0s', format_time(0))
Assert("format_time second", "1s", format_time(1)) Assert('format_time second', '1s', format_time(1))
Assert("format_time decasecond", "10s", format_time(10)) Assert('format_time decasecond', '10s', format_time(10))
Assert("format_time minute", "1m", format_time(60)) Assert('format_time minute', '1m', format_time(60))
Assert("format_time milliday", "1m 26s", format_time(86)) # missing 0.4s due to precision Assert('format_time milliday', '1m 26s', format_time(86)) # missing 0.4s due to precision
Assert("format_time hectosecond", "1m 40s", format_time(100)) Assert('format_time hectosecond', '1m 40s', format_time(100))
Assert("format_time kilosecond", "16m 40s", format_time(int(1e3))) Assert('format_time kilosecond', '16m 40s', format_time(int(1e3)))
Assert("format_time hour", "1h", format_time(3600)) Assert('format_time hour', '1h', format_time(3600))
Assert("format_time day", "1j", format_time(86400)) Assert('format_time day', '1j', format_time(86400))
Assert("format_time week/sennight", "1w", format_time(604800)) Assert('format_time week/sennight', '1w', format_time(604800))
Assert("format_time megasecond", "1w 4j 13h 46m 40s", format_time(int(1e6))) Assert('format_time megasecond', '1w 4j 13h 46m 40s', format_time(int(1e6)))
Assert("format_time fortnight", "2w", format_time(1209600)) Assert('format_time fortnight', '2w', format_time(1209600))
Assert("format_time lunar month (draconitic)", "3w 6j 5h 5m 35s", format_time(2351135)) # missing 0.8 due to precision Assert('format_time lunar month (draconitic)', '3w 6j 5h 5m 35s', format_time(2351135)) # missing 0.8 due to precision
Assert("format_time lunar month (tropical)", "3w 6j 7h 43m 4s", format_time(2360584)) # missing 0.7 due to precision Assert('format_time lunar month (tropical)', '3w 6j 7h 43m 4s', format_time(2360584)) # missing 0.7 due to precision
Assert("format_time lunar month (sidereal)", "3w 6j 7h 43m 11s", format_time(2360591)) # missing 0.6 to precision Assert('format_time lunar month (sidereal)', '3w 6j 7h 43m 11s', format_time(2360591)) # missing 0.6 to precision
Assert("format_time lunar month (anomalistic)", "3w 6j 13h 18m 33s", format_time(2380713)) # missing 0.2 due to precision Assert('format_time lunar month (anomalistic)', '3w 6j 13h 18m 33s', format_time(2380713)) # missing 0.2 due to precision
Assert("format_time lunar month (synodic)", "4w 1j 12h 44m 2s", format_time(2551442)) # missing 0.9 due to precision Assert('format_time lunar month (synodic)', '4w 1j 12h 44m 2s', format_time(2551442)) # missing 0.9 due to precision
Assert("format_time month", "1M", format_time(2678400)) Assert('format_time month', '1M', format_time(2678400))
Assert("format_time quarantine", "1M 1w 2j", format_time(int(3456e3))) Assert('format_time quarantine', '1M 1w 2j', format_time(int(3456e3)))
Assert("format_time semester", "4M 2j", format_time(10886400)) Assert('format_time semester', '4M 2j', format_time(10886400))
Assert("format_time lunar year", "11M 1w 6j 8h 52m 48s", format_time(30617568)) Assert('format_time lunar year', '11M 1w 6j 8h 52m 48s', format_time(30617568))
Assert("format_time year", "1y", format_time(int(31536e3))) Assert('format_time year', '1y', format_time(int(31536e3)))
Assert("format_time tropical year", "1y 5h 48m 45s", format_time(31556925)) # missing 0.216 due to precision Assert('format_time tropical year', '1y 5h 48m 45s', format_time(31556925)) # missing 0.216 due to precision
Assert("format_time gregorian year", "1y 5h 49m 12s", format_time(31556952)) Assert('format_time gregorian year', '1y 5h 49m 12s', format_time(31556952))
Assert("format_time sidereal year", "1y 6h 9m 9s", format_time(31558149)) # missing 0.7635456 due to precision Assert('format_time sidereal year', '1y 6h 9m 9s', format_time(31558149)) # missing 0.7635456 due to precision
Assert("format_time leap year", "1y 1j", format_time(31622400)) Assert('format_time leap year', '1y 1j', format_time(31622400))
Assert("format_time olympiad", "4y", format_time(int(126144e3))) Assert('format_time olympiad', '4y', format_time(int(126144e3)))
Assert("format_time lusturm", "5y", format_time(int(15768e4))) Assert('format_time lusturm', '5y', format_time(int(15768e4)))
Assert("format_time decade", "10y", format_time(int(31536e4))) Assert('format_time decade', '10y', format_time(int(31536e4)))
Assert("format_time indiction", "15y", format_time(int(47304e4))) Assert('format_time indiction', '15y', format_time(int(47304e4)))
Assert("format_time score", "20y", format_time(int(63072e4))) Assert('format_time score', '20y', format_time(int(63072e4)))
Assert("format_time gigasecond", "31y 8M 1w 4j 1h 46m 40s", format_time(int(1e9))) Assert('format_time gigasecond', '31y 8M 1w 4j 1h 46m 40s', format_time(int(1e9)))
Assert("format_time jubilee", "50y", format_time(int(15768e5))) Assert('format_time jubilee', '50y', format_time(int(15768e5)))
Assert("format_time century", "1c", format_time(int(31536e5))) Assert('format_time century', '1c', format_time(int(31536e5)))
Assert("format_time millennium", "10c", format_time(int(31536e6))) Assert('format_time millennium', '10c', format_time(int(31536e6)))
Assert("format_time age", "257c 72y", format_time(int(812745792e3))) Assert('format_time age', '257c 72y', format_time(int(812745792e3)))
Assert("format_time terasecond", "3170c 97y 10M 3w 4j 17h 46m 40s", format_time(int(1e13))) Assert('format_time terasecond', '3170c 97y 10M 3w 4j 17h 46m 40s', format_time(int(1e13)))
Assert("format_time megaannum", "10000c", format_time(int(31536e9))) Assert('format_time megaannum', '10000c', format_time(int(31536e9)))
Assert("format_time petasecond", "317097c 91y 11M 2w 4j 1h 46m 40s", format_time(int(1e15))) Assert('format_time petasecond', '317097c 91y 11M 2w 4j 1h 46m 40s', format_time(int(1e15)))
Assert("format_time galactic year", "2300000c", format_time(int(725328e10))) Assert('format_time galactic year', '2300000c', format_time(int(725328e10)))
Assert("format_time eon", "10000000c", format_time(int(31536e12))) Assert('format_time eon', '10000000c', format_time(int(31536e12)))
Assert("format_time kalpa", "43200000c", format_time(int(13623552e10))) Assert('format_time kalpa', '43200000c', format_time(int(13623552e10)))
Assert("format_time exasecond", "317097919c 83y 9M 1h 46m 40s", format_time(int(1e18))) Assert('format_time exasecond', '317097919c 83y 9M 1h 46m 40s', format_time(int(1e18)))
# Cannot use number bigger than currently supported ISO Python # Cannot use number bigger than currently supported ISO Python
#Assert("format_time zettasecond", "", format_time(1e21)) # Assert('format_time zettasecond', '', format_time(1e21))
#Assert("format_time yottasecond", "", format_time(1e24)) # Assert('format_time yottasecond', '', format_time(1e24))
#Assert("format_time ronnasecond", "", format_time(1e27)) # Assert('format_time ronnasecond', '', format_time(1e27))
#Assert("format_time quettasecond", "", format_time(1e30)) # Assert('format_time quettasecond', '', format_time(1e30))
# uint64_t_MAX == 2**64 == 18446744073709551615 == -1 # uint64_t_MAX == 2**64 == 18446744073709551615 == -1
Assert("format_time max", "5849424173c 55y 3w 5j 7h 16s", format_time(int(2**64 - 1))) Assert('format_time max', '5849424173c 55y 3w 5j 7h 16s', format_time(int(2**64 - 1)))
def format_time_ns_test() -> None: def format_time_ns_test() -> None:
"""Test suite for the format_time_ns output """Test suite for the format_time_ns output
See https://en.wikipedia.org/wiki/Unit_of_time for details See https://en.wikipedia.org/wiki/Unit_of_time for details
""" """
Assert("format_time_ns null", "0ns", format_time_ns(0)) Assert('format_time_ns null', '0ns', format_time_ns(0))
Assert("format_time_ns nanosecond", "1ns", format_time_ns(1)) Assert('format_time_ns nanosecond', '1ns', format_time_ns(1))
Assert("format_time_ns shake", "10ns", format_time_ns(10)) Assert('format_time_ns shake', '10ns', format_time_ns(10))
Assert("format_time_ns microsecond", "1µs", format_time_ns(int(1e3))) Assert('format_time_ns microsecond', '1µs', format_time_ns(int(1e3)))
Assert("format_time_ns millisecond", "1ms", format_time_ns(int(1e6))) Assert('format_time_ns millisecond', '1ms', format_time_ns(int(1e6)))
Assert("format_time_ns centisecond", "10ms", format_time_ns(int(1e7))) Assert('format_time_ns centisecond', '10ms', format_time_ns(int(1e7)))
Assert("format_time_ns decisecond", "100ms", format_time_ns(int(1e8))) Assert('format_time_ns decisecond', '100ms', format_time_ns(int(1e8)))
Assert("format_time_ns second", "1s", format_time_ns(int(1e9))) Assert('format_time_ns second', '1s', format_time_ns(int(1e9)))
Assert("format_time_ns decasecond", "10s", format_time_ns(int(1e10))) Assert('format_time_ns decasecond', '10s', format_time_ns(int(1e10)))
Assert("format_time_ns minute", "1m", format_time_ns(int(6e10))) Assert('format_time_ns minute', '1m', format_time_ns(int(6e10)))
Assert("format_time_ns milliday", "1m 26s 400ms", format_time_ns(int(864e8))) Assert('format_time_ns milliday', '1m 26s 400ms', format_time_ns(int(864e8)))
Assert("format_time_ns hectosecond", "1m 40s", format_time_ns(int(1e11))) Assert('format_time_ns hectosecond', '1m 40s', format_time_ns(int(1e11)))
Assert("format_time_ns kilosecond", "16m 40s", format_time_ns(int(1e12))) Assert('format_time_ns kilosecond', '16m 40s', format_time_ns(int(1e12)))
Assert("format_time_ns hour", "1h", format_time_ns(int(36e11))) Assert('format_time_ns hour', '1h', format_time_ns(int(36e11)))
Assert("format_time_ns day", "1j", format_time_ns(int(864e11))) Assert('format_time_ns day', '1j', format_time_ns(int(864e11)))
Assert("format_time_ns week/sennight", "1w", format_time_ns(int(6048e11))) Assert('format_time_ns week/sennight', '1w', format_time_ns(int(6048e11)))
Assert("format_time_ns megasecond", "1w 4j 13h 46m 40s", format_time_ns(int(1e15))) Assert('format_time_ns megasecond', '1w 4j 13h 46m 40s', format_time_ns(int(1e15)))
Assert("format_time_ns fortnight", "2w", format_time_ns(int(12096e11))) Assert('format_time_ns fortnight', '2w', format_time_ns(int(12096e11)))
Assert("format_time_ns lunar month (draconitic)", "3w 6j 5h 5m 35s 800ms", format_time_ns(int(23511358e8))) Assert('format_time_ns lunar month (draconitic)', '3w 6j 5h 5m 35s 800ms', format_time_ns(int(23511358e8)))
Assert("format_time_ns lunar month (tropical)", "3w 6j 7h 43m 4s 700ms", format_time_ns(int(23605847e8))) Assert('format_time_ns lunar month (tropical)', '3w 6j 7h 43m 4s 700ms', format_time_ns(int(23605847e8)))
Assert("format_time_ns lunar month (sidereal)", "3w 6j 7h 43m 11s 600ms", format_time_ns(int(23605916e8))) Assert('format_time_ns lunar month (sidereal)', '3w 6j 7h 43m 11s 600ms', format_time_ns(int(23605916e8)))
Assert("format_time_ns lunar month (anomalistic)", "3w 6j 13h 18m 33s 200ms", format_time_ns(int(23807132e8))) Assert('format_time_ns lunar month (anomalistic)', '3w 6j 13h 18m 33s 200ms', format_time_ns(int(23807132e8)))
Assert("format_time_ns lunar month (synodic)", "4w 1j 12h 44m 2s 900ms", format_time_ns(int(25514429e8))) Assert('format_time_ns lunar month (synodic)', '4w 1j 12h 44m 2s 900ms', format_time_ns(int(25514429e8)))
Assert("format_time_ns month", "1M", format_time_ns(int(26784e11))) Assert('format_time_ns month', '1M', format_time_ns(int(26784e11)))
Assert("format_time_ns quarantine", "1M 1w 2j", format_time_ns(int(3456e12))) Assert('format_time_ns quarantine', '1M 1w 2j', format_time_ns(int(3456e12)))
Assert("format_time_ns semester", "4M 2j", format_time_ns(int(108864e11))) Assert('format_time_ns semester', '4M 2j', format_time_ns(int(108864e11)))
Assert("format_time_ns lunar year", "11M 1w 6j 8h 52m 48s", format_time_ns(int(30617568e9))) Assert('format_time_ns lunar year', '11M 1w 6j 8h 52m 48s', format_time_ns(int(30617568e9)))
Assert("format_time_ns year", "1y", format_time_ns(int(31536e12))) Assert('format_time_ns year', '1y', format_time_ns(int(31536e12)))
Assert("format_time_ns tropical year", "1y 5h 48m 45s 216ms", format_time_ns(int(31556925216e6))) Assert('format_time_ns tropical year', '1y 5h 48m 45s 216ms', format_time_ns(int(31556925216e6)))
Assert("format_time_ns gregorian year", "1y 5h 49m 12s", format_time_ns(int(31556952e9))) Assert('format_time_ns gregorian year', '1y 5h 49m 12s', format_time_ns(int(31556952e9)))
Assert("format_time_ns sidereal year", "1y 6h 9m 9s 763ms 545µs 600ns", format_time_ns(int(315581497635456e2))) Assert('format_time_ns sidereal year', '1y 6h 9m 9s 763ms 545µs 600ns', format_time_ns(int(315581497635456e2)))
Assert("format_time_ns leap year", "1y 1j", format_time_ns(int(316224e11))) Assert('format_time_ns leap year', '1y 1j', format_time_ns(int(316224e11)))
Assert("format_time_ns olympiad", "4y", format_time_ns(int(126144e12))) Assert('format_time_ns olympiad', '4y', format_time_ns(int(126144e12)))
Assert("format_time_ns lusturm", "5y", format_time_ns(int(15768e13))) Assert('format_time_ns lusturm', '5y', format_time_ns(int(15768e13)))
Assert("format_time_ns decade", "10y", format_time_ns(int(31536e13))) Assert('format_time_ns decade', '10y', format_time_ns(int(31536e13)))
Assert("format_time_ns indiction", "15y", format_time_ns(int(47304e13))) Assert('format_time_ns indiction', '15y', format_time_ns(int(47304e13)))
Assert("format_time_ns score", "20y", format_time_ns(int(63072e13))) Assert('format_time_ns score', '20y', format_time_ns(int(63072e13)))
Assert("format_time_ns gigasecond", "31y 8M 1w 4j 1h 46m 40s", format_time_ns(int(1e18))) Assert('format_time_ns gigasecond', '31y 8M 1w 4j 1h 46m 40s', format_time_ns(int(1e18)))
Assert("format_time_ns jubilee", "50y", format_time_ns(int(15768e14))) Assert('format_time_ns jubilee', '50y', format_time_ns(int(15768e14)))
Assert("format_time_ns century", "1c", format_time_ns(int(31536e14))) Assert('format_time_ns century', '1c', format_time_ns(int(31536e14)))
Assert("format_time_ns millennium", "10c", format_time_ns(int(31536e15))) # Python int too large to convert to C long
Assert("format_time_ns age", "257c 72y", format_time_ns(int(812745792e12))) # Assert('format_time_ns millennium', '10c', format_time_ns(int(31536e15)))
Assert("format_time_ns terasecond", "3170c 97y 10M 3w 4j 17h 46m 40s", format_time_ns(int(1e22))) # Assert('format_time_ns age', '257c 72y', format_time_ns(int(812745792e12)))
Assert("format_time_ns megaannum", "10000c", format_time_ns(int(31536e18))) # Assert('format_time_ns terasecond', '3170c 97y 10M 3w 4j 17h 46m 40s', format_time_ns(int(1e22)))
# Assert('format_time_ns megaannum', '10000c', format_time_ns(int(31536e18)))
# Cannot use number bigger than currently supported ISO Python # Cannot use number bigger than currently supported ISO Python
# Assert("format_time_ns petasecond", "317097c 91y 11M 2w 4j 1h 46m 40s", format_time_ns(int(1e24))) # Assert('format_time_ns petasecond', '317097c 91y 11M 2w 4j 1h 46m 40s', format_time_ns(int(1e24)))
# Assert("format_time_ns galactic year", "2300000c", format_time_ns(int(725328e19))) # Assert('format_time_ns galactic year', '2300000c', format_time_ns(int(725328e19)))
# Assert("format_time_ns eon", "10000000c", format_time_ns(int(31536e21))) # Assert('format_time_ns eon', '10000000c', format_time_ns(int(31536e21)))
# Assert("format_time_ns kalpa", "43200000c", format_time_ns(int(13623552e19))) # Assert('format_time_ns kalpa', '43200000c', format_time_ns(int(13623552e19)))
# Assert("format_time_ns exasecond", "317097919c 83y 9M 1h 46m 40s", format_time_ns(int(1e27))) # Assert('format_time_ns exasecond', '317097919c 83y 9M 1h 46m 40s', format_time_ns(int(1e27)))
# Assert("format_time_ns zettasecond", "", format_time_ns(int(1e30))) # Assert('format_time_ns zettasecond', '', format_time_ns(int(1e30)))
# Assert("format_time_ns yottasecond", "", format_time_ns(int(1e33))) # Assert('format_time_ns yottasecond', '', format_time_ns(int(1e33)))
# Assert("format_time_ns ronnasecond", "", format_time_ns(int(1e36))) # Assert('format_time_ns ronnasecond', '', format_time_ns(int(1e36)))
# Assert("format_time_ns quettasecond", "", format_time_ns(int(1e39))) # Assert('format_time_ns quettasecond', '', format_time_ns(int(1e39)))
# uint64_t_MAX == 2**64 == 18446744073709551615 == -1 # uint64_t_MAX == 2**64 == 18446744073709551615 == -1
Assert("format_time_ns max", "5c 84y 11M 2j 23h 34m 33s 709ms 551µs 615ns", format_time_ns(2**64 - 1)) Assert('format_time_ns max', '5c 84y 11M 2j 23h 34m 33s 709ms 551µs 615ns', format_time_ns(2**64 - 1))